climate-mirror / datasets

For tracking data mirroring progress

201 stars 18 forks source link

Atmospheric Data Research Center Langley (ASDC DAAC) #301

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago

ftp://l5eil01.larc.nasa.gov

Not sure how much, if any, of this directory is replicated elsewhere, but I can imagine that these datasets are at high risk

erikfriesen commented 7 years ago

going to try to get a file size estimate

erikfriesen commented 7 years ago

Either this is already cleared out, or the server isn't returning much. Will someone else please have a look and give me a sanity check?

donbright commented 7 years ago

Parts of this are described in the following issues

177 ftp:/l5eil01.larc.nasa.gov/SAGE_III

175 ftp:/l5eil01.larc.nasa.gov/ACRIM_III/ACR3L2DM.001

176 ftp:/l5eil01.larc.nasa.gov/ACRIM_III/ACR3L2SC.001

80 https://eosweb.larc.nasa.gov/projects-supported-list

donbright commented 7 years ago

@erikfriesen it is actually full of data, its just a twisty maze of symbolic links which is why the size reports so weird.

example SAGE_III is full of symbolic links to /distribution/SAGE_III which is full of big files

erikfriesen commented 7 years ago

ah, i see. du -h just finished. It estimated 144TB. Not sure why my first run of it returned so little.

JeremiahCurtis commented 7 years ago

well, I have nowhere near 144 TB of storage, so here's hoping someone else has a good connection and lots of space

It could be broken up

I've started downloading ftp://l5eil01.larc.nasa.gov/VBBE/ but am still slogging through ftp://l5eil01.larc.nasa.gov/VBBE/VBEMI2AE.002 This could take awhile, as I'm also running projects for NCDC and NCEI issues that have not yet been claimed, and my connection is pretty slow, even with ethernet

gabefair commented 7 years ago

I am breaking this ticket up into parts, standby for new ticket. Please post under: #337, containing tickets to all other subsets

gabefair commented 7 years ago

/VBBE/ is being tracked at #349