Dataset at ftp://ghrc.nsstc.nasa.gov/pub/data

climate-mirror / datasets

For tracking data mirroring progress

201 stars 18 forks source link

Dataset at ftp://ghrc.nsstc.nasa.gov/pub/data #168

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/ghrc.nsstc.nasa.gov/pub/data.

Suggested in a large email containing many urls

clickbg commented 7 years ago

~~Downloading~~

clickbg commented 7 years ago

Giving up on this one as it is over 547G

donbright commented 7 years ago

Size estimates

lftp -c 'open ftp://ghrc.nsstc.nasa.gov/pub/data && du -h -d1'
8.8G    ./aces                                                                 
602M    ./ampr                                                             
330G    ./ksc-fieldmill                                                        
13G     ./ksc-ldar                                                             
826M    ./lip                                                         
137M    ./ols                                                                  
58G     ./otd                                                           
72G     ./SANDS                                                                
66G     ./tmi-op                                                              
547G    total

bkirkbri commented 7 years ago

According to ftp://ghrc.nsstc.nasa.gov/pub/data/ksc-fieldmill/doc/kscmill_dataset.html:

This dataset contains measurements of the electric field strength at each sensor of the KSC Advanced Ground Based Field Mill (AGBFM) network. The AGBFM network consists of 34 field mills of which, as of 05/29/97, only 31 are presently working. These data are used in real time at KSC in the Launch Pad Lightning Warning System (LPLWS).

That is unlikely to be at risk data. Excluding it reduces the size of this dataset greatly.

gabefair commented 7 years ago

Maybe we should break this one up into several issue tickets. Atmospheric pollutants have an effect on lightning strikes. I could imagine a motivation to kill this dataset

StephWo commented 7 years ago

Downloading to Offline-Mirror for now

jradzuweit commented 7 years ago

Started to buildup mirror at ftp://176.9.40.3/mirror_a

jradzuweit commented 7 years ago

Mirror available at ftp://176.9.40.3/mirror_a; hashdeep just running will be available soon

StephWo commented 7 years ago

@jradzuweit you were quite quick with that download. what commands did you use?

jradzuweit commented 7 years ago

I use

wget -c --mirror --wait=5 \<dataset>

but my machine has a 1GBit interface, this is good for this kind of job. The wait should be put always in to give the server a chance to open connection, otherwise you would consume all the bandwidth and the server is not reachable for others.

StephWo commented 7 years ago

@jradzuweit In fact our machines performance and bandwidth should be almost identical, certainly our connection quality is. We use the same service provider for our root-servers as it seems :) Your IP-Range seemed familiar. Thats why I was surprised that you were so quick. Thanks for the info

StephWo commented 7 years ago

I've got this one downloaded to an offline mirror. I will create a torrent at some point in the future, but that might take a while

entr0p1 commented 7 years ago

Grabbing this

entr0p1 commented 7 years ago

Done, finally!

Checksums: https://gateway.ipfs.io/ipfs/QmdvTXjiDxAC9pJhmbzebZPVxtE1JomoYa6gPDXJP5BiCo Root Directory: https://gateway.ipfs.io/ipfs/QmarbRUBFscr7pcApEfurUBN7h4TmyCTPEh1Ptv8UvkLJY Size: 587GB