search

e-alfred / ocdownloader

ocDownloader - AGPL-licensed multi-protocol download manager for Nextcloud using ARIA2, youtube-dl and Curl (supports Youtube, BitTorrent, HTTP, FTP)
https://github.com/e-alfred/ocdownloader
GNU Affero General Public License v3.0
375 stars 85 forks source link

set referer and useragent #28

Closed DJCrashdummy closed 7 years ago

DJCrashdummy commented 7 years ago

i found this issue at the old project (https://github.com/DjazzLab/ocdownloader/issues/24) and for admins this would be a useful setting.

Nibbels commented 7 years ago

Yes this ist definitly something that a hoster might check and block.

For webbots you normally just copy the string from some new firefox or chrome. Another Header would be: "HTTP_ACCEPT" as "text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8", "/", "text/html, application/xhtml+xml, /", "" "HTTP_ACCEPT_CHARSET" as empty or "utf-8,iso-8859-1,utf-16,*;q=0.001". "HTTP_ACCEPT_ENCODING" as "gzip, deflate" but the downloader has to understand what it tells the source :D "HTTP_ACCEPT_LANGUAGE" as "de-DE, en-US" or "de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" "HTTP_USER_AGENT" as "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36 OPR/47.0.2631.71 (Edition Campaign 75)" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0" so on.

e-alfred commented 7 years ago

This would be a useful feature. But I don't see a possibility to dynamically set the user agent for Aria2c. It only allows to set the user agent for HTTP(S) using the an additional parameter:

https://aria2.github.io/manual/en/html/aria2c.html?highlight=user%20agent#cmdoption-U

For now, it could be added to the readme file on how to set up Aria2 with a custom user agent. For Curl, an additional setting in the admin settings could be integrated, but it would be very confusing because both downloaders would have different configuration methods for the user agent then.

DJCrashdummy commented 7 years ago

the same is valid for referer: https://aria2.github.io/manual/en/html/aria2c.html?highlight=referer#cmdoption--referer

well, we can integrate nearly the whole man-page of aria2c into the readme, but that wouldn't make much sense... just kidding. i guess people who understand why there are 2 different backends and know how to handle them, also know what to do with a man-page; although a small hint (maybe with an example) is never wrong. :wink:

if it isn't possible to change the useragent and referer for aria2c via the GUI, how are the proxy-settigs achieved?

and if it still isn't possible, i don't see such a big thing with settings only for cURL... there are also settings just for BitTorrent or Youtube. with a hint, that all (other) settings for aria2c must be done via the command line for the daemon, would be perfectly fine (for now).

@e-alfred what do you mean with "dynamically set the user agent"? - maybe we talk past each other...? i'm talking about a "simple" setting for the whole instance via admin-setting (ideally in the GUI for all backends).

e-alfred commented 7 years ago

@DJCrashdummy By dynamically I mean setting it for each seperate download instead of having one fixed setting in the admin section. This is more useful because different download pages could require different user agents/referers.

I integrated the options to set a referer and user agent in the download section for both http and ftp downloads. Please test it here: https://github.com/e-alfred/ocdownloader/pull/34

Sample output from Apache logs:

server - - [17/Sep/2017:15:21:26 +0200] "GET /gallery/file.jpg HTTP/1.1" 200 45280 "http://google.com" "testtmp"

e-alfred commented 7 years ago

Implemented in https://github.com/e-alfred/ocdownloader/pull/34

You can set a user agent and referer in the download section below the link input field for both HTTP/FTP.