climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp://podaac-ftp.jpl.nasa.gov/allData/climatology/ #288

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago
axlecrusher commented 7 years ago

lftp podaac-ftp.jpl.nasa.gov:/allData/climatology> du -h 0 ./L3 59K ./retired/L3/docs 5.0K ./retired/L3/pathfinder_climatology/9km/docs 2.0K ./retired/L3/pathfinder_climatology/9km/sw 156M ./retired/L3/pathfinder_climatology/9km 156M ./retired/L3/pathfinder_climatology 1.6G ./retired/L3/pathfinder_Version5.0/8day/day 1.6G ./retired/L3/pathfinder_Version5.0/8day/night 3.2G ./retired/L3/pathfinder_Version5.0/8day 369M ./retired/L3/pathfinder_Version5.0/monthly/day 390M ./retired/L3/pathfinder_Version5.0/monthly/night 758M ./retired/L3/pathfinder_Version5.0/monthly 3.9G ./retired/L3/pathfinder_Version5.0 82M ./retired/L3/pf_nsipp_5day/9km 82M ./retired/L3/pf_nsipp_5day 9.7M ./retired/L3/pf_nsipp_month/9km 9.7M ./retired/L3/pf_nsipp_month 23K ./retired/L3/sw 4.2G ./retired/L3 4.2G ./retired 4.2G .

axlecrusher commented 7 years ago

Downloading

axlecrusher commented 7 years ago

Completed download of 253MB. Not sure why its so much smaller.

wantonwonton commented 7 years ago

I would think some files are probably missing. Did you use wget for the download? It defaults to limiting recursion depth to 5 levels. You can use the -m/--mirror switch to select more appropriate settings.

axlecrusher commented 7 years ago

Thank you. I had forgot to use -m and used -r instead.

axlecrusher commented 7 years ago

Done.

If anyone wants to verify here is the md5sum of the contents find podaac-ftp.jpl.nasa.gov/allData/climatology -type f -exec md5sum {} \; | sort -k 2 | md5sum f659995ef9706b1cc2ad50a444a9c9da -

erikfriesen commented 7 years ago

downloading a copy. will check md5 when complete

erikfriesen commented 7 years ago

Grabbed all the data, the size looks correct.

Having a problem with the md5sum though (i am new at this)

I have the files stored in /data/climate

When i run find climate/ -type f -exec md5sum {} \; | sort -k 2 | md5sum from within /data, i get d7fe7cb0f83b380d4cf7689eb2aaeb51

When i run find . -type f -exec md5sum {} \; | sort -k 2 | md5sum from within /data/climate, i get 763abd82c0784eddda8abc1464e04c96

erikfriesen commented 7 years ago

I can't seem to make the md5sum match, but i have downloaded it twice now (and maintained the original directory structure this time instead of dumping it into a directory named "climate") and the md5sums between those two match.

erikfriesen commented 7 years ago

find podaac-ftp.jpl.nasa.gov/allData/climatology -type f -exec md5sum {} \; | sort -k 2 | md5sum

6610bc1829075746bfe56105eedd8342

bkirkbri commented 7 years ago

@axlecrusher @erikfriesen Do you have URLs for these mirrors? Thanks!

erikfriesen commented 7 years ago

@bkirkbri i don't have it on a public facing mirror yet. still need to get that setup. the data is on my home server right now.

axlecrusher commented 7 years ago

@erikfriesen I agree with your md5sum

find podaac-ftp.jpl.nasa.gov/allData/climatology/ -type f -exec md5sum {} \; | grep -v '.listing' | sort -k 2 | md5sum 6610bc1829075746bfe56105eedd8342 -

ymarcus93 commented 7 years ago

Ok, I made a torrent for this. Please let me know if magnet link is working. I also found out more about this data set.

Description: Contains three data sets. (1): Monthly and pentad (five day) climatologies of sea surface temperature (SST) data using the V4, V4.1, and interim V4.1 NOAA/NASA AVHRR Oceans Pathfinder data. (2) A high resolution SST climatology has been generated from daily 9 km Advanced Very High Resolution Radiometer (AVHRR) Pathfinder SST satellite data through Gaussian interpolation and (3) pixel-by-pixel averaging approach. See retired/L3/docs/*.html in archive for more information on project. Last modified 6/6/15...seems to be retired. Size: 4.35GB

Magnet: magnet:?xt=urn:btih:f4aceee1d370ccdb665f32c7feff9b31ea787d46&dn=podaac-ftp.jpl.nasa.gov%5Fclimatology.tar.gz&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=udp%3A%2F%2F9.rarbg.com%3A2710&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337)

SHA256 Hash: df2dff3345411ca72c24157b510d254ec961b84f3c3bad9f596f42a4d5aa056d

isaacnooni commented 5 years ago

Hello, I kindly need a step by step guidance on how to process and estimate TWS using GRACE or GLDAS soil moisture. Thanks