climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NASA PODAAC JASON-1 Satellite Data #262

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/podaac.jpl.nasa.gov/allData/jason1.

Suggested in a large email containing many urls

adinbied commented 7 years ago

I'm on it!

siennathesane commented 7 years ago

What does this data set measure?

adinbied commented 7 years ago

From NASA: "Jason-1 was the first follow-on to the highly successful TOPEX/Poseidon mission. This second joint NASA-CNES mission was built on a French spacecraft, and launched on an American Delta II rocket from Vandenburg Air Force Base in California. Like TOPEX/Poseidon, the payload included both American and French instruments. Jason-1 altimeter data is part of a suite of data provided by other JPL-managed ocean missions. The GRACE mission uses two satellites to accurately measure Earth's gravity field (mass distribution), and the QuikSCAT scatterometer mission provided almost 11 years of ocean-surface winds. Jason-1 was decommissioned in July 2013, and continued the task of providing the important oceanographic data time-series originated by TOPEX/Poseidon for 11-1/2 years."

aqyx commented 7 years ago

On it, downloading to local storage ATM. Appears to be sea level information (temp, density, etc.) from the JASON-1 satellite.

EDIT: 153GB currently queued, it appears to be much larger than that.

adinbied commented 7 years ago

Still downloading, should I abort?

siennathesane commented 7 years ago

ideally two mirrors/copies are good, but only one is necessary, your call.

adinbied commented 7 years ago

My main concern is the size, aqyx said it would probably be much larger than 153 GB, and I'm copying using wget, so I don't know how big the final result will be. I'll probably keep it going for another couple hours, at least.

aqyx commented 7 years ago

220GB currently queued, and it is absolutely nowhere near finished processing.

siennathesane commented 7 years ago

I have a few terabytes I can lend, if need be. 💾

aqyx commented 7 years ago

OK, I've cancelled my download. I wasn't expecting it to be anywhere near as big as this - likely well over a terabyte.

gabefair commented 7 years ago

@aqyx good thinking. A complete dataset is better than an incomplete dataset posing as complete

adinbied commented 7 years ago

I've cancelled mine as well. Hopefully someone with a fast connection and lots of space can pick it up.

siennathesane commented 7 years ago

I'll start the download. Thanks guys. :)

siennathesane commented 7 years ago

I've been continuously downloading for 2 days, and I'm only at 5GB...

alex-kazda commented 7 years ago

If I'm right that this is the same repository as #194, then the total size of just L1 and L2 subdirectories is about 8.8 TB.

lftp podaac-ftp.jpl.nasa.gov:/allData/jason1> du -h -d1 .
28G     ./L1                                                                   
8.8T    ./L2                                                           
5.1G    ./preview
3.6T    ./retired
13T .
108111 commented 7 years ago

I've been downloading for 30 hours, downloading L1 of podaac-ftp.jpl.nasa.gov:/allData/jason1 under #194 (which @alex-kazda pointed out is the same).

Currently up to 'L1/.snapshot/hourly.2/rinex/trsr1810.07o’

However, I'm at 24GB already. The 28GB size doesn't include /.snapshot as it is a hidden folder.

Is anyone else downloading this? @mxplusb how are you doing? If not, I'll carry on. I don't have room for /L2, /preview and /retired right now, but going to see if I can borrow some more storage, I have 6TB free at the minute.

So, do I stop to download something else or continue?

gabefair commented 7 years ago

Maybe we should break up this one into different issue tickets?

108111 commented 7 years ago

@gabefair agree, borrowed 7TB so downloading /L1, /preview and /retired.

Suggest new ticket for L2 (I don't know how to, sorry!)

bkirkbri commented 7 years ago

Strange that L1 would be smaller than L2 since it's the latter is derived from the former.

bkirkbri commented 7 years ago

@108111 Did you manage to get /L1?

x775 commented 7 years ago

I am going through L1 and L2 as of this posting.

Will update once done.