climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp://podaac-ftp.jpl.nasa.gov/allData/seawifs/ #293

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago

ftp://podaac-ftp.jpl.nasa.gov/allData/seawifs/

bkirkbri commented 7 years ago

This appears to be 9.1GB.

markuslaker commented 7 years ago

Downloading now:

wget --output-file=log.txt --mirror --no-verbose ftp://podaac-ftp.jpl.nasa.gov/allData/seawifs/

I'll figure out how to make a Torrent file from this data tomorrow and post the result here. (Yep, I'm a torrent newbie.)

markuslaker commented 7 years ago

We're downloading at about 300MB/hr, and so, if bkirkbri's estimate of 9.1GB is correct, it'll be Thursday before I can finish downloading and produce a Torrent.

donbright commented 7 years ago

attempting download for offline archive.

markuslaker commented 7 years ago

wget segfaulted early this morning after downloading about 3GiB. I'll try again this evening, using lftp.

sveeke commented 7 years ago

Downloading it right now. When done, it will be available on http://climate-mirror-1.i11.nl.

donbright commented 7 years ago

offline copy complete. 9.1G

hashdeep: offline, contact me if data is needed


updated hashdeep 2/2 thanks @axlecrusher (see below notes)

axlecrusher commented 7 years ago

Mirroring

markuslaker commented 7 years ago

Now available as a torrent. Github won't let me attach a .torrent file directly, and so I've zipped it.

This is the first torrent file I've created, so I'd appreciate some feedback telling me whether it works. Many thanks.

Edit: deleted the obsolete torrent file. Instead, please use the magnet link below.

axlecrusher commented 7 years ago

@markuslaker That torrent zip file extracts strangely. You could post a magnet link for the torrent file.

markuslaker commented 7 years ago

Thanks, @axlecrusher. I found a service called torrent2magnet.com, which yielded this magnet link:

Edit: removed the obsolete magnet link, which no longer works. Please use the new magnet link below.

axlecrusher commented 7 years ago

@markuslaker For some reason torrent2magnet.com leaves off the tracker part of the mirror link. Some bittorrent clients let you click on the torrent and copy the magnet link.

Though that being said, the magnet seemed to work.

markuslaker commented 7 years ago

I've installed a new BitTorrent client that supports magnet links natively. Here's a new magnet link:

magnet:?xt=urn:btih:R36WPS67MJQJR4C6J2FEYQJHXHFBAJXD&dn=SeaViewingWideFieldOfViewSensor(SeaWIFS)1997-2010.zip&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.leechers-paradise.org%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.zer0day.to%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=http%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=http%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2710%2fannounce&tr=udp%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=http%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=http%3a%2f%2ftracker.dler.org%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.internetwarriors.net%3a1337%2fannounce&tr=udp%3a%2f%2ftracker1.wasabii.com.tw%3a6969%2fannounce&tr=udp%3a%2f%2ftracker2.wasabii.com.tw%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.mg64.net%3a6969%2fannounce&tr=udp%3a%2f%2fmgtracker.org%3a6969%2fannounce&tr=udp%3a%2f%2fmgtracker.org%3a2710%2fannounce&tr=udp%3a%2f%2fipv4.tracker.harry.lu%3a80%2fannounce&tr=http%3a%2f%2ftracker1.wasabii.com.tw%3a6969%2fannounce&tr=http%3a%2f%2fmgtracker.org%3a6969%2fannounce&tr=http%3a%2f%2ftracker2.wasabii.com.tw%3a6969%2fannounce&tr=http%3a%2f%2ftracker.sktorrent.net%3a6969%2fannounce&tr=http%3a%2f%2ftracker.mg64.net%3a6881%2fannounce&tr=http%3a%2f%2fipv4.tracker.harry.lu%3a80%2fannounce&tr=udp%3a%2f%2ftracker.vanitycore.co%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.baravik.org%3a6970%2fannounce&tr=udp%3a%2f%2fbt.xxx-tracker.com%3a2710%2fannounce&tr=http%3a%2f%2ftracker.vanitycore.co%3a6969%2fannounce&tr=http%3a%2f%2ftracker.baravik.org%3a6970%2fannounce&tr=udp%3a%2f%2ftracker.kamigami.org%3a2710%2fannounce&tr=udp%3a%2f%2ftracker.grepler.com%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.filetracker.pl%3a8089%2fannounce&tr=udp%3a%2f%2ftracker.tiny-vps.com%3a6969%2fannounce&tr=http%3a%2f%2ftracker.grepler.com%3a6969%2fannounce&tr=http%3a%2f%2ftracker.filetracker.pl%3a8089%2fannounce&tr=http%3a%2f%2ftracker.tiny-vps.com%3a6969%2fannounce&tr=http%3a%2f

axlecrusher commented 7 years ago

@donbright my hashdeep does not match. I will try to re-download the broken files.

hashdeep -l -r -a -vv -k ../hashdeep.audit.txt ./allData/seawifs ./allData/seawifs/L3/chlA/9km/monthly/2004/S20042752004305.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/2004/S20040012004031.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/2003/S20031212003151.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/1999/S19991521999181.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/2002/S20020912002120.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/2000/S20000922000121.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used ./allData/seawifs/L3/chlA/9km/monthly/2001/S20011522001181.L3m_MO_CHL_chlor_a_9km.bz2: Known file not used hashdeep: Audit failed Input files examined: 0 Known files expecting: 0 Files matched: 9313 Files partially matched: 0 Files moved: 0 New files found: 7 Known files not found: 7

axlecrusher commented 7 years ago

@donbright You might need to re-download those files. The error messages remained after I re-downloaded.

axlecrusher commented 7 years ago

md5sum of everything

find allData/seawifs/ -type f -exec md5sum {} \; | grep -v \\.listing | sort -k 2 | md5sum 3b79d769bcfa3e40b512a27f57f817f4 -

donbright commented 7 years ago

@axlecrusher thank you i appreciate that... my hash matches yours now. woohoo

3b79d769bcfa3e40b512a27f57f817f4

will update my hashdeep asap.