search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp://podaac-ftp.jpl.nasa.gov/allData/seawinds/ #297

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago

ftp://podaac-ftp.jpl.nasa.gov/allData/seawinds/

erikfriesen commented 7 years ago

checking file size now. will update when done or if it's taking an exceptionally long time.

erikfriesen commented 7 years ago

file size check still going. going to go ahead and put it on hold for now.

axlecrusher commented 7 years ago

Any size estimate?

erikfriesen commented 7 years ago

@axlecrusher: I don't have a really good one. The count was up above 200GB when I cut it off.

markuslaker commented 7 years ago

I'll update this message as I find the sizes of the rest of the subdirectories. Given daily directories going back to 2003, this will be an overnight job at least, but several instances of lftp are on the case.

L2B          6.6GiB
L2B12       25GiB
L2BE         6.5GiB
L3
L3E           1.4GiB
README.txt
ancillary     8.5GiB
preview       0
retired       7.5GiB
rustyguts commented 7 years ago

Downloading now with GCS.

kliu128 commented 7 years ago

Fully downloaded, with five files/directories failed (seem to be broken on the server):

At a rough guess, it's at least 1 terabyte, probably more like 1.5-1.8 TB.

Mirror is accessible at https://climate.archives.potatofrom.space/podaac-ftp.jpl.nasa.gov/allData/seawinds. Hashdeep is in progress and will be at https://climate.archives.potatofrom.space/podaac%5Cx2dftp.jpl.nasa.gov-allData-seawinds.hashdeep.txt. I'll edit this post when the hashdeep is complete.

EDIT: Hashdeep is complete. SHA256SUM of hashdeep file: d9b8c5ee86b549611e04203e25050180541876fe329bf8d0a69e3dff8f3fd4bf.

bkirkbri commented 7 years ago

@Pneumaticat Thank you. Calculating total size from your hashdeep file now. FYI, digests of hashdeep files are of limited use because relative pathnames may be different or a single extra file would render the digest completely different. You probably know that, but I'm commenting for posterity. @RustyGuts Were you able to complete this one?

bkirkbri commented 7 years ago

Total size of files in @Pneumaticat hashdeep file is 836GB. Would you check on it for me?

kliu128 commented 7 years ago

@bkirkbri Yeah, du -sh gives ~839 GB—seems like 1.5–1.8 TB was a bit of an overestimation on my part. And good point about the digest, although it's still useful to ensure download integrity, I suppose ;)

gabefair commented 7 years ago

downloading now

rustyguts commented 7 years ago

Compressed mirror available here https://cinder7.org/datasets/climate-mirror-datasets/297-podaac-subset-seawinds/

@gabefair