search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp:/cddis.gsfc.nasa.gov/slr/data #150

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/cddis.gsfc.nasa.gov/slr/data.

Suggested in a large email containing many urls

kliu128 commented 7 years ago

Downloading now! Seems to be ~160 GB.

donbright commented 7 years ago

rough size estimate: 171 GB using lftp du -a

kliu128 commented 7 years ago

Download complete! I guess it'll remain as a private mirror for now, although I can probably get it up on a web server (with fairly poor availability, mind).

kliu128 commented 7 years ago

Got it up at https://climate.archives.potatofrom.space/cddis.gsfc.nasa.gov/slr/data/. (Uptime is not guaranteed.)

bkirkbri commented 7 years ago

@Pneumaticat can you provide a hashdeep -erl ./cddis.gsfc.nasa.gov/slr/data for cross verification?

Ichimonji10 commented 7 years ago

Downloading for cross verification purposes.

$ wget \
  --mirror \
  --output-file=cddis.gsfc.nasa.gov.log \
  --no-verbose \
  --limit-rate=2m \
  ftp://cddis.gsfc.nasa.gov/slr/data/

It'll take a couple days, due to limited bandwidth.

kliu128 commented 7 years ago

@bkirkbri hashdeep up at https://climate.archives.potatofrom.space/cddis.gsfc.nasa.gov-slr-data.hashdeep.txt.

Ichimonji10 commented 7 years ago

Downloaded ~165GB of data. Hashdeep and download log available at: http://www.ichimonji10.name/climatemirror/cddis.gsfc.nasa.gov.slr_data/

bkirkbri commented 7 years ago

Comparing the two hashdeep files shows a number of mismatches. Could be added/removed files over the time period between your mirrors. I'm downloading now to post another mirror.

bkirkbri commented 7 years ago

Mirror: https://s3.ca-central-1.amazonaws.com/climate-mirror/cddis.gsfc.nasa.gov/slr/data/index.html Hashdeep: https://s3.ca-central-1.amazonaws.com/climate-mirror/cddis.gsfc.nasa.gov/cddis.gsfc.nasa.gov_slr_data.hashdeep.audit.txt