search

helgee / RemoteFiles.jl

Download files from the Internet and keep them up-to-date.
Other
49 stars 8 forks source link

Allow other backends to download the files #8

Closed ronisbr closed 5 years ago

ronisbr commented 6 years ago

Hi @helgee ,

For some reason, sometimes I cannot download fluxtable.txt using curl but I can using wget. The address is ftp://ftp.geolab.nrcan.gc.ca/data/solar_flux/daily_flux_values/fluxtable.txt. Hence, I am wondering if it is possible to let the user select which backend they want to download the files.

helgee commented 6 years ago

Sounds like a good idea. I would also like to have the ability to use HTTP.jl. We are having a PDR this week so I will not get to it before next week.

ronisbr commented 6 years ago

Nice! Good luck with the PDR! I will try to submit a PR so that you can take a look next week.

bgodard commented 5 years ago

I cannot retrieve file via ftp using curl here, but this is likely due to the local firewall.

$ curl ftp://ftp.geolab.nrcan.gc.ca/data/solar_flux/daily_flux_values/fluxtable.txt
curl: (13) Bad PASV/EPSV response: 502

However it works if I use active mode ftp instead of passive mode by passing the correct option to curl:

curl -P - ftp://ftp.geolab.nrcan.gc.ca/data/solar_flux/daily_flux_values/fluxtable.txt

or with wget, since wget will try passive mode, fail and then try active mode and succeed.

Maybe one of the following options would fix all these problems: 1) if this is an ftp connection, try active ftp if passive ftp fails: use curl -P - on the second try. 2) if curl fails, retry with wget.

Reference: https://discourse.julialang.org/t/examples-of-analysis-that-can-be-done-with-satellitetoolbox-jl/19330/7?u=bernard_godard

helgee commented 5 years ago

I have implemented a simple backend system. Please have a look at master or a new version that is coming in the next few days.