climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NASA PODAAC JASON-2 Satellite Data #252

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Correct, full dataset at ftp://ftp.nodc.noaa.gov/pub/data.nodc/jason2/

Several directories are symlinks. Be sure that your tool mirrors them. For lftp use the -L flag to mirror.

PLEASE IGNORE ORIGINAL URL ftp:/podaac-ftp.jpl.nasa.gov/allData/ostm

Suggested in a large email containing many urls

TwistTheNeil commented 7 years ago

I can grab this one if it hasn't already

TwistTheNeil commented 7 years ago

I have downloaded the set and will have an offline backup (for now)!

ftp podaac-ftp.jpl.nasa.gov:/allData> mirror --parallel=10 ostm Total: 565 directories, 100771 files, 128 symlinks New: 100771 files, 128 symlinks 26317908448 bytes transferred in 258546 seconds (99.4 KiB/s) lftp podaac-ftp.jpl.nasa.gov:/allData> exit neil@riot:~/climate/podaac-ftp.jpl.nasa.gov$ du -sh 25G .

bkirkbri commented 7 years ago

It seems that there is a different location with more JASON-2 data, see https://podaac.jpl.nasa.gov/forum/viewtopic.php?f=72&t=487

bkirkbri commented 7 years ago

Confirmed that full data is at NOAA. FTP access is available at ftp://ftp.nodc.noaa.gov/pub/data.nodc/jason2/

lftp du shows sizes as

3.2T    ./gdr -> /nodc/data/jason2-xgdr/gdr
3.0T    ./igdr -> /nodc/data/jason2-xgdr/igdr
446G    ./ogdr -> /nodc/data/jason2-xgdr/ogdr
50G     ./current
30M     ./documentation
914M    ./gridded_cycle_mean
0       ./orders
170M    ./QA_assurance
96M     ./QA_assurance_old
19G     ./subs

So around 6-7TB total

bkirkbri commented 7 years ago

@TwistTheNeil Thank you for grabbing the initial URL. I've update this issue with the larger, full dataset. Hopefully someone has room for that.

gabefair commented 7 years ago

This one needs to be broken up

tkyocum commented 7 years ago

I have all of this. If anyone's got the space, I'm happy to share. It's about 4TB compressed/9TB uncompressed.