search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NASA PODAAC QuickScat Data #265

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/podaac.jpl.nasa.gov/allData/quikscat.

Suggested in a large email containing many urls

gabefair commented 7 years ago

Downloading now

erikfriesen commented 7 years ago

Getting ready to queue this one up for download

bkirkbri commented 7 years ago

@erikfriesen @gabefair Please update when you have complete mirrors. Thanks!

erikfriesen commented 7 years ago

@gabefair: how large did the data set turn out to be? Still waiting on the results of a du -h from within lftp. A bit worried that it may be too large for my machine.

gabefair commented 7 years ago

This one is massive. Every time the FTP finishes, I rerun it with -N -m to double check and it finds more to download. I'm currently at 90gbs

erikfriesen commented 7 years ago

@gabefair: Understood. Please let me know when you end up with a final size. Going to move on to another data set for now, but I'll keep an eye on this one.

bkirkbri commented 7 years ago

I'm estimating the size of this one now.

bkirkbri commented 7 years ago

Size according to lftp du:

6.0T    L1B
21G     L1C
825G    L2A
4.0T    L2A12
127G    L2B
1.9T    L2B12
8.1T    L3
1.8G    ancillary
50G     preview
104G    retired

So, at least 13TB total. I will update when L3 estimate is complete...

bkirkbri commented 7 years ago

L3 estimate is still running five days later. I can only imagine how big this one is...

bkirkbri commented 7 years ago

L3 estimate still running! After sampling subdirectories and extrapolating I estimate that L3 is around 10TB in size.

bkirkbri commented 7 years ago

Finally finished L3 is 8.1TB.

entr0p1 commented 7 years ago

Have downloaded this - podaac.jpl.nasa.gov/allData/quikscat/preview

Checksums: https://gateway.ipfs.io/ipfs/Qmbopj5tTGT6DHB3mWzFFV9ijehvbTCmgWZP12h4vqRA57 Root Directory: https://gateway.ipfs.io/ipfs/QmSFppBNF8mmkLPjH3YMyZqMAxqNZ2tH8fgU58btnRfhRu Size: 50GB

entr0p1 commented 7 years ago

Downloaded this - podaac.jpl.nasa.gov/allData/quickscat/L1C

Checksums: https://gateway.ipfs.io/ipfs/QmUHK7LyDz74vFpWmDvrwrmoELNMg85E4WY2JuKWxtaaLh Root Directory: https://gateway.ipfs.io/ipfs/QmQN68UayBSJCet6AVmQryh1ic3JS5HriWgZ1gEnHGTyVV Size: 22GB