search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NOAA NCEI Ocean Winds #277

Open gabefair opened 7 years ago

gabefair commented 7 years ago

DO NOT START, SEE OTHER TICKETS BELOW

OceanWinds

/OceanWinds/ has the following children: aquarius/ 2/25/13, 12:00:00 AM : addressed in #236 ascat/ 5/7/13, 12:00:00 AM : duplicate of #237 ccmp/ 5/7/13, 12:00:00 AM : duplicate of #239 monthly_mean_atlas/ 5/7/13, 12:00:00 AM : duplicate of #327 nimbus7/ 10/25/11, 12:00:00 AM : duplicate of #326 nscat/ 5/10/13, 12:00:00 AM : duplicate of #263 & #196 oceansat2/ 5/10/13, 12:00:00 AM : duplicate of #264 poet_ifremer/ 4/25/12, 12:00:00 AM : addressed in #330 quikscat/ 6/22/16, 12:00:00 AM : duplicate of #265 rapidscat/ 5/14/16, 12:00:00 AM : duplicate of #266 seasat/ 5/10/13, 12:00:00 AM : duplicate of #209 seawinds/ 5/10/13, 12:00:00 AM : duplicate of #297 ssmi/ 5/10/13, 12:00:00 AM : duplicate of #295 windsat/ 7/29/13, 12:00:00 AM : duplicate of #296 woce/ 5/7/13, 12:00:00 AM : duplicate of #325

Once you are done, please share:

We are keeping this ticket open for logistical reasons until the child tickets are closed.

baobrien commented 7 years ago

Downloading now.

gabefair commented 7 years ago

@baobrien This one is massive. Hows it coming? We can break it up if it would be better managed in smaller chuncks

gabefair commented 7 years ago

Did you use command: wget -N -m ftp://podaac.jpl.nasa.gov/OceanWinds/* ? The -N and -m will resume where you left off if you have to restart. We want to make sure we got the entire dataset.

k15z commented 7 years ago

I'm downloading it too, it's going slowly. Currently working on aquarius, ascat, and ccmp - downloaded ~200 GB so far.

provos commented 7 years ago

Getting a copy of this, too.

JeremiahCurtis commented 7 years ago

Are the above subfolders identical to ftp://podaac-ftp.jpl.nasa.gov/allData/seawinds/ and ftp://podaac-ftp.jpl.nasa.gov/allData/windsat/ ?

bkirkbri commented 7 years ago

Many of the directories in ftp://podaac.jpl.nasa.gov/OceanWinds seem to overlap with /allData/... which is broken into many issues.

gabefair commented 7 years ago

@provos , @k15z how is it coming along? Can you provide us links to your dataset once you are done? Thanks

provos commented 7 years ago

At 345G for allData/aquarius right now and 145G on allData/modis. I hope the 3TB I have for serving will be enough.

provos commented 7 years ago

At 490G for aquarius and 450G for modis.

bkirkbri commented 7 years ago

It does seem like /OceanWinds is symlinked into ../allData and duplicates the issues above.

@provos Did you fetch from ftp://podaac.jpl.nasa.gov/allData/aquarius rather than the issue URL of ftp://podaac.jpl.nasa.gov/OceanWinds/aquarius? If so, will you post completed mirror info in #236 as that issue is more specific?

bkirkbri commented 7 years ago

aquarius consists only of symlinks into ../allData/aquarius and is definitely a duplicate of #236

bkirkbri commented 7 years ago

@gabefair Thanks for tracking down the duplicates. I've verified that all content is a duplicate (creating a few new issues that didn't exist).

Anyone having a partial mirror of this issue should consult the duplicates list at the top and post their complete submirrors in the appropriate issue. Thank you!