DFO-Ocean-Navigator / netcdf-timestamp-mapper

Maps timestamps (and variables) to a corresponding nc file using sqlite3.
https://dfo-ocean-navigator.github.io/netcdf-timestamp-mapper/
GNU General Public License v3.0
1 stars 1 forks source link

Implement remote crawling #15

Open htmlboss opened 4 years ago

htmlboss commented 4 years ago

We have access to FTP sites that could be indexed in order to display even more data in the Navigator without having to copy anything. Thing is, the c++ file system library does not support network-ed URLs. We need to hook into libcurl (which is already present in the conda environment due to the netcdf library) to create a temporary session and grab a list of the files present at the URL.