climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp://nssdcftp.gsfc.nasa.gov/pub/data/wind #185

Closed nickrsan closed 7 years ago

nickrsan commented 7 years ago

ftp://nssdcftp.gsfc.nasa.gov/pub/data/wind

Suggested in a large email containing many urls

as-com commented 7 years ago

Not found

bofh453 commented 7 years ago

Sigh, so every single link of the form ftp://nssdcftp.gsfc.nasa.gov/spacecraft_data/$SPACECRAFT is actually at ftp://nssdcftp.gsfc.nasa.gov/pub/data/$SPACECRAFT

Backup in progress as of a minute ago.

gabefair commented 7 years ago

EDIT: I resend my vote to close as there was initial confusion as to which dataset this issue ticket was for.

donbright commented 7 years ago

size estimates with @bohf453 's alternate url

lftp -c 'open ftp://nssdcftp.gsfc.nasa.gov/pub/data/wind && du -h -d1'
121G    ./3dp                                                                  
4.2G    ./epact                                                                
497G    ./mfi                                                                  
1.4G    ./orbit                                                                
151G    ./swe                                                                  
77M     ./swics                                                       
102G    ./waves                                                                
870M    ./windnrt                                                         
877G    total
bkirkbri commented 7 years ago

@gabefair Why do you vote to close? Is this a duplicate or not relevant? Thanks!

bkirkbri commented 7 years ago

@bofh453 Were you able to get a mirror of this one?

StephWo commented 7 years ago

Downloading...

StephWo commented 7 years ago

mirror available at http://176.9.83.61/185 hashdeep file at http://176.9.83.61/185/185_hashdeep.txt (HUGE!)

total dataset-size: 878 GB or 920475212 Byte

bkirkbri commented 7 years ago

@BauerPiepenbrink thank you!

gabefair commented 7 years ago

I have a complete copy of this data set. It updates daily, so I might have more files than everyone else.

https://www.dropbox.com/sh/1itpf5nwwwlp609/AAAiE-YoFWabPtGKJZR00p5Ea?dl=0 Command used: wget -N -m ftp://nssdcftp.gsfc.nasa.gov/pub/data/wind/* FINISHED --2017-02-24 23:20:38--

Computing md5deep now using: md5deep -erl https://www.dropbox.com/s/5y3mfsehc8k1hum/wind_md5_audit.txt?dl=0

Results of du -b --max-depth=1 --human-readable: 122G ./3dp 162G ./swe 4.2G ./epact 1.4G ./orbit 499G ./mfi 76M ./swics 102G ./waves 876M ./windnrt 890G ./

EDIT: I have updated with the correct file sizes

StephWo commented 7 years ago

@gabefair I cross checked with your md5 sums (http://176.9.83.61/185/wind_diff_gabefair.txt) and all the files you have are in my archive as well (except the .listing, but that doesn't matter)

on the other hand, if I check the folder size like you did, I get:

122G ./3dp 151G ./swe 4,2G ./epact 1,4G ./orbit 498G ./mfi 79M ./swics 102G ./waves 873M ./windnrt 878G .

So either there are files missing in your archive or I followed a bunch of symlinks while downloading and have the same files a few times.

However, you could cross check with my hashdeep file to determine why we have different sizes: http://176.9.83.61/185/185_hashdeep.txt

gabefair commented 7 years ago

Thanks for checking this. We have the same files, nothing is missing. I just learned that du will report the "size on disk" and since I use NTFS compression, my numbers were not actual sizes. I will remove them now and update my other tickets to show the correct size.

Fyi use -b to get real sizes du -b --max-depth=1 --human-readable or ls -l

gabefair commented 7 years ago

1 vote to close

We have it mirrored twice in this ticket

StephWo commented 6 years ago

Be advised: because of changes in my hardware demands I wont be able to host this or the other datasets any longer after April 2018. Please create a copy if necessary before the end of April. The Full list of Dataset Issue-Numbers that are mirrored on my server and will not be hosted after April:

162 175 176 184 185 279 291 362

Find all these datasets at http://176.9.83.62 or http://climatemirror1.space