search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp://ftp.ngdc.noaa.gov/ #292

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago

ftp://ftp.ngdc.noaa.gov/

wantonwonton commented 7 years ago

I've got an lftp du running on this (for 8 hours so far...). (Unfortunately, I didn't use flags that would give information on individual directory trees.)

gabefair commented 7 years ago

We need to break this one up into other Issue tickets on github

sveeke commented 7 years ago

@wantonwonton do you know more about the size?

wantonwonton commented 7 years ago

@sveeke My Internet connection went down for a while this morning, causing the du to terminate. The incomplete total that was reported was 1.7TB. Some of it may not be relevant data, though. I recall seeing some directories related to seismicity. Sorry I don't have any more details.

ivanstegic commented 7 years ago

I'm going to start on this, and we'll see where we get!

donbright commented 7 years ago

ongoing Size estimates, going folder by folder based on the main INDEX.txt file in /. excluding /pub "public area" which had no descriptor for now. will update as 'du' command progresses. 


76G DMSP/          Defense Meteorological Satellite Data Archive

3G geomag/        Geomagnetism and geomagnetics models

58G hazards/       Natural Hazards data, volcanoes, tsunamis, earthquakes

0  international/ International program information on IAGA/Oersted/wdc

>1.7T ionosonde/     Ionosonde data (note - very rough size estimate, du stopped working)

761k mgg/           Limited Marine Geology and Geophysics (most data in http area)
associated website: https://www.ngdc.noaa.gov/mgg/mggd.html

0 Snow_Ice/      Snow and Ice Data Center 

328M Solid_Earth/   Historic Solid Earth Geophysics (cm note  - TerrainBase 94)
associated website: http://www.ngdc.noaa.gov/seg/segd.shtml

623G   STP/           Solar-Terrestrial Physics

53G      wdc/           World Data Service for Geophysics, formerly World Data Centers

it should be noted that a good deal of this type of NOAA data appears to be accessed through very sophisticated web applications , such as https://maps.ngdc.noaa.gov/viewers/bathymetry/

i am not sure how much of that is going to be missed, if this FTP server is the only thing copied.

ivanstegic commented 7 years ago

I'm at 400GB, but had to stop for a few days. I'm back at this now with vigor. It will be available permanently here: http://data.ourdataourhands.org/ftp.ngdc.noaa.gov/