Closed elidana closed 2 years ago
I don't have any obvious answers to these issues. My go to method no longer works either.
For GNSS sitelogs, another possible (still not ideal, but less painful) option is to first list all available in a specific location, and then do a loop over the file list.
The example below is for someone only interested to download the most recent sitelogs available for each GeoNet continuous GNSS station which code is starting with "A"
#/bin/bash
http=https://data.geonet.org.nz/gnss/sitelogs/logs
curl -l ${http}/ | grep -o 'href=".*">' | sed 's/href="//;s/\/">//;s/">//' > tmp.$$
for sta in `grep "^a" tmp.$$ | cut -c 1-4 | sort -u`; do
file=`grep $sta tmp.$$ | tail -1`
curl -O ${http}/${path}/${file}
done
rm tmp.$$
Hi,
I'm also in the process of changing our workflow from FTP to HTTP. We previously used Python parsing of wget's FTP .listing
files to see which files are available on the server, compare their size and last-modified date to what we had locally, and if they didn't match, re-download the file.
For the first part, seeing which files are available, we now download the html page in Python and use regex patterns to parse all the available filenames. Not very elegant, since the index page layout can change with http server updates, so I'm interested to see whether more robust approaches are gonna pop up here.
For the second part, the checking whether remote files are different from local files, also based on the output of the http index page, it seems to me now that looking at some random days (e.g., https://data.geonet.org.nz/gnss/rinex/2022/001/), that all files have apparently been modified in the last two days - including files from 1999? Is there currently some reprocessing going on and should I redownload all the files, or is there something happening on the server side that changes the last-modified timestamps for all the files? In the latter case, we will have to rely on only comparing the file sizes, which of cours ealso works, but is not 100% robust if the changes are only minor.
Thanks!
Hello @tobiscode ,
thanks a lot for your input and providing this useful background on your workflow!
Hopefully what you find below will answer your two issues nicely.
Very recently, we have started providing our GeoNet GNSS data via a second (and hopefully more efficient) access mechanism.
Data are now available via an Amazon Web Service storage solution: AWS S3 bucket.
Details are available from https://registry.opendata.aws/geonet/
This is not fully advertised yet on our website as we are still in the process of populating the S3 bucket with additional datasets, but all GNSS data (except https://data.geonet.org.nz/gnss/rinex1Hz/) are now available via https (https://data.geonet.org.nz/gnss) and via AWS (
s3://geonet-open-data/gnss`).
With an AWS Client, you can easily sync the GNSS data to your local storage, and only download the updated files.
Python has some AWS client wrapper as well you might want to check out, if you are interested (we use boto3
).
If interested and if you have further questions on that, please feel free to open a new "issue" here, and I'll be more than happy to follow up.
In terms of robust approaches to download data from an HTTPs server, I'd be keen to know if other users have better ideas, we haven't found any, but I'm very pleased to see this conversation and to receive your comment on that. For the file timestamp, I suspect that the updated timestamp is not reliable, and only due to the fact that the host that serve the HTTPs repository has a cache on its backend, that gets refreshed every so often (we had some unrelated issues a couple of weeks ago, and all the cache was refreshed, that might be why you see files from 1999 being updated).
Real file updates will happen when:
Hi,
thanks for your reply!
With regard to the HTTPS, it does seem like the 1999 data was updated again yesterday, so yes it does seem to be unreliable, since I'm guessing none of your bullet points were the case for that data since my last post. (It's still puzzling though, I'm interfacing like this with other servers and they don't seem to have that issue... But I totally understand that that's not a high-priority issue.)
I just tried to install boto3
in the system that we use to download data and it seems that our system is too old, even using virtual environments, it didn't work out-of-the-box. That's a problem on our side though, and I'll see if/when we can upgrade our system to be able to use the S3 option, but until then, I guess we're stuck with HTTPS.
Either way, thanks for clearing things up!
Cheers
Thanks for checking that out @tobiscode,
sorry to know your system cannot handle boto3
. Another option is to use the aws cli in a bash code, the documentation is available here https://docs.aws.amazon.com/cli/latest/reference/s3/, but I suspect that you might bump into a similar issue there too.
And yes, I confirm that we are not updating the 1999 files. Our own setup might be different from other data centres, that might be why you see differences.
Thanks to you for your interest and comment!
Elisabetta
Hi,
just as a follow-up, conda came to the rescue and I was able to set up Python with boto3
on our old machine - it's working flawlessly so far, and on top of being able to compare last-modified timestamps, this new way is much faster than both FTP or HTTPS. Thanks for the help, and setting up the S3 storage in the first place!
Cheers
oh that is fantastic news, thanks a lot @tobiscode for this feedback!
We have received some similar questions from end users, on how to optimize data retieval from the HTTPs GeoNet data repository.
I am copying those request here so that we can point other users to this reply if needed.
user n.1:
user n.2: