search

sissbruecker / linkding

Self-hosted bookmark manager that is designed be to be minimal, fast, and easy to set up using Docker.
MIT License
5.33k stars 261 forks source link

User Agent, or, Plans for LD_SINGLEFILE_OPTIONS #690

Closed pettijohn closed 2 months ago

pettijohn commented 3 months ago

I see you stubbed out this config variable LD_SINGLEFILE_OPTIONS but aren't using it. One use case I have for it is passing user agent:

Does not work: single-file "https://reddit.com/" r.html

Yields (truncated output): 

<body>
 <h1>whoa there, pardner!</h1>
<p>Your request has been blocked due to a network policy.</p>
<p>Additionally make sure your User-Agent is not empty and is something unique and descriptive and try again.</p>

Works: single-file "https://reddit.com/" --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0" r2.html

Before I start work on it for a pull request, do you have any design requirements? Any specific arguments you want promoted to their own environment variable?

Ref https://github.com/sissbruecker/linkding/blob/edd958fff61a640cf02c7640b717e78f10322b8b/bookmarks/services/singlefile.py#L20-L23

sissbruecker commented 3 months ago

I was using args = [singlefile_path, singlefile_options, url, temp_filepath] but that failed when calling subprocess.Popen. Did not investigate further then and just removed it. Could be that the argument must not be empty. Could also be that if the option contains multiple arguments that this requires splitting into separate argument strings.

I guess for now it would be enough if people could dump all single-file arguments that they want to use into that option like --arg1 value1 --arg2 value2 ... and singlefile.py just makes it work. User agent is a good point though, I'm a bit surprised that single-file doesn't set one by default. Would make sense to introduce a default in linkding maybe, though that can be done after making custom settings work.

pettijohn commented 3 months ago

I can test (e.g. with empty environment variable) and submit a pull request later, but this works in the basic case:

Docker compose: - LD_SINGLEFILE_OPTIONS=--user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0"

singlefile.py

import shlex

singlefile_path = settings.LD_SINGLEFILE_PATH
singlefile_options = shlex.split(settings.LD_SINGLEFILE_OPTIONS) # parses string to list of arguments
temp_filepath = filepath + ".tmp"

args = [singlefile_path] + singlefile_options + [url, temp_filepath] # concat lists
sissbruecker commented 3 months ago

Looks good 👍. I can add some basic tests if you open a PR.