search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NASA Earth Science Satellites #26

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Name: NASA Earth Science Satellites Organization: NASA Description URL: http://daac.gsfc.nasa.gov Download URL: File Types: Size: Status:

johncronan commented 7 years ago

An option for download URL: https://atrain.gesdisc.eosdis.nasa.gov/data/

Uses HTTP Auth for file downloads (but not index pages). You need to have a NASA Earthdata login.

Their notes on bulk downloads: https://disc.sci.gsfc.nasa.gov/recipes/?q=recipes/How-to-Download-Data-Files-from-HTTP-Service-with-wget

johncronan commented 7 years ago

Turns out atrain is just part of Goddard Earth Sciences. Here are the other URLs:

https://acdisc.gesdisc.eosdis.nasa.gov/data/ https://gpm1.gesdisc.eosdis.nasa.gov/data/ https://goldsmr4.gesdisc.eosdis.nasa.gov/data/ https://measures.gesdisc.eosdis.nasa.gov/data/ https://oco2.gesdisc.eosdis.nasa.gov/data/ https://disc2.gesdisc.eosdis.nasa.gov/data/

stephanne-t commented 7 years ago

I am currently working on downloading this (offline mirror, for now).

stephanne-t commented 7 years ago

I think I have this entirely downloaded (including the links listed in the previous comment) but will verify this tomorrow. It's on an offline mirror currently, looks like it's ~8 TB.

stephanne-t commented 7 years ago

I have https://atrain.gesdisc.eosdis.nasa.gov/data/AMSRE, but some of the other stopped partway through (possibly tried to do too many at once). It's up to 12 TB total, will try and get them one at a time, hopefully without issues.

stephanne-t commented 7 years ago

This is massive, and I can't do all of it. (One product for the MODIS instrument on one satellite is nearly 10 TB, and there are nearly 20 products for that instrument alone). I'm currently working in seeing what has been scraped in other issues, and what still needs scraping.