search

zaidka / cliget

A Firefox addon to download login-protected files from the command line
https://addons.mozilla.org/en-US/firefox/addon/cliget/
Mozilla Public License 2.0
357 stars 57 forks source link

TrueFX.com redirects to index.html (wget, curl, aria2c). Firefox downloads ok. #40

Open webradio opened 7 years ago

webradio commented 7 years ago

TrueFX.com requires a free registration, after that .zip downloads are possible. I'm using Firefox and can download files in a usual way, e.g. http://truefx.com/dev/data/2009/NOVEMBER-2009/AUDJPY-2009-11.zip

wget, curl and aria2c all get redirected to index.html, as it would happen in the browser if I weren't signed in to truefx.com and simply tried http://truefx.com/dev/data/2009/NOVEMBER-2009/AUDJPY-2009-11.zip directly.

Below are command lines produced by cliget. (I only shortened cookies with my user id etc before posting it here). For comparison, "Free Download Manager" can download this file when I use it's Firefox add-on. Sure enough, it also gets redirected to index.html when I try adding the URL from above directly.

Any clues where could TrueFX "feel" the difefrence between a browser and wget/curl/aria with cliget command line?

d:\TrueFX>wget --header="Host: truefx.com" --header="Use r-Agent: Mozilla/5.0 (Windows NT 6.1; rv:48.0) Gecko/20100101 Firefox/48.0" --he ader="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" - -header="Accept-Language: de,en-US;q=0.7,en;q=0.3" --header="Content-Type: appli cation/x-www-form-urlencoded" --header="Cookie: uid=e9f; oivey=rememberme; _ga=GA1.2.114.147; persistence=%2F %3Fpage%3Ddownload%26description%3Dnovember%26dir%3D2009%2FNOVEMBER-2009" "http: //truefx.com/dev/data/2009/NOVEMBER-2009/EURCHF-2009-11.zip" -c

d:\TrueFX\curl>curl --header "Host: truefx.com" --header "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:48.0) Gecko/20100101 Firefox/48.0" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0 .8" --header "Accept-Language: de,en-US;q=0.7,en;q=0.3" --header "Accept-Encodin g: gzip, deflate" --header "Content-Type: application/x-www-form-urlencoded" --h eader "Cookie: uid=e9f; oivey=rememberme; _ ga=GA1.2.114.147; persistence=%2F%3Fpage%3Ddownload%26description% 3Dnovember%26dir%3D2009%2FNOVEMBER-2009; _gat=1" "http://truefx.com/dev/data/200 9/NOVEMBER-2009/AUDJPY-2009-11.zip" -O -J -L

d:\TrueFX\aria2>aria2c --header "Host: truefx.com" --hea der "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:48.0) Gecko/20100101 Firefox/48 .0" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*; q=0.8" --header "Accept-Language: de,en-US;q=0.7,en;q=0.3" --header "Accept-Enco ding: gzip, deflate" --header "Content-Type: application/x-www-form-urlencoded" --header "Cookie: uid=e9f7; oivey=rememberme ; _ga=GA1.2.114.147; persistence=%2F%3Fpage%3Ddownload%26descripti on%3Dnovember%26dir%3D2009%2FNOVEMBER-2009; _gat=1" "http://truefx.com/dev/data/ 2009/NOVEMBER-2009/AUDJPY-2009-11.zip" --async-dns=false --continue=true --file- allocation=none --max-connection-per-server=1

nogweii commented 7 years ago

My guess is that truefx is checking the Referer header.