search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

ftp://podaac-ftp.jpl.nasa.gov/allData/windsat/ #296

Open JeremiahCurtis opened 7 years ago

JeremiahCurtis commented 7 years ago

ftp://podaac-ftp.jpl.nasa.gov/allData/windsat/

bkirkbri commented 7 years ago

I estimate this is about 21GB based on 3-6MB per day for 13 years.

erikfriesen commented 7 years ago

~Running du-h via lftp now. will update with file size and start downloading if it's small enough my my machine.~

Actually, I'm already hammering this server with another lftp du -h, i'll leave it be until that one finishes.

bkirkbri commented 7 years ago

It's so many directories and small files that I ran du on a wide sample, found pretty consistent results and extrapolated.

erikfriesen commented 7 years ago

Cool, I'm gonna trust your extrapolation and just start downloading since the du -h is still running

erikfriesen commented 7 years ago

Download complete. find podaac-ftp.jpl.nasa.gov/allData/windsat/ -type f -exec md5sum {} \; | sort -k 2 | md5sum returns 0785732d2d84369ef5c6cee16a10f961

axlecrusher commented 7 years ago

What was the size?

erikfriesen commented 7 years ago

Looks like it's about 17GB

markuslaker commented 7 years ago

Mirroring now. I'll produce a torrent when it's done.

axlecrusher commented 7 years ago

Mirroring

axlecrusher commented 7 years ago

@markuslaker If you make the torrent directly from the mirrored directory structure (without zipping up the entire directory structure), I think anyone who has mirrored the source could mirror your torrent without needing to download the entire torrent again. Though before doing so, we would want to verify that the source has been mirrored correctly via hashes.

markuslaker commented 7 years ago

@axlecrusher, thanks for the hint. I'll do that.

axlecrusher commented 7 years ago

Completed though my md5sum is the following

find podaac-ftp.jpl.nasa.gov/allData/windsat/ -type f -exec md5sum {} \;| grep -v \\.listing | sort -k 2 | md5sum df311d07ebf6e13fb18498255a9ae164 -

axlecrusher commented 7 years ago

I just noticed that this directory is still being updated daily. That would account for the md5sum mismatch.

markuslaker commented 7 years ago

It took a while to download, but the magnet link is below. It covers a directory tree, as @axlecrusher suggested; I've not zipped up the data.

No one has used the magnet link I posted under Issue 293, and so I'm not sure I'm doing any real good here. Is it possible to get these magnet links posted somewhere more visible, so that the data can start to be distributed more widely?

magnet:?xt=urn:btih:BMAXR6SCAV7ACQ3SFKQTECXYRDSKKC4K&dn=ftp_podaac-ftp_jpl_nasa_gov_allData_windsat&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.leechers-paradise.org%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.zer0day.to%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=http%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=http%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2710%2fannounce&tr=udp%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=http%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=http%3a%2f%2ftracker.dler.org%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.internetwarriors.net%3a1337%2fannounce&tr=udp%3a%2f%2ftracker1.wasabii.com.tw%3a6969%2fannounce&tr=udp%3a%2f%2ftracker2.wasabii.com.tw%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.mg64.net%3a6969%2fannounce&tr=udp%3a%2f%2fmgtracker.org%3a6969%2fannounce&tr=udp%3a%2f%2fmgtracker.org%3a2710%2fannounce&tr=udp%3a%2f%2fipv4.tracker.harry.lu%3a80%2fannounce&tr=http%3a%2f%2ftracker1.wasabii.com.tw%3a6969%2fannounce&tr=http%3a%2f%2fmgtracker.org%3a6969%2fannounce&tr=http%3a%2f%2ftracker2.wasabii.com.tw%3a6969%2fannounce&tr=http%3a%2f%2ftracker.sktorrent.net%3a6969%2fannounce&tr=http%3a%2f%2ftracker.mg64.net%3a6881%2fannounce&tr=http%3a%2f%2fipv4.tracker.harry.lu%3a80%2fannounce&tr=udp%3a%2f%2ftracker.vanitycore.co%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.baravik.org%3a6970%2fannounce&tr=udp%3a%2f%2fbt.xxx-tracker.com%3a2710%2fannounce&tr=http%3a%2f%2ftracker.vanitycore.co%3a6969%2fannounce&tr=http%3a%2f%2ftracker.baravik.org%3a6970%2fannounce&tr=udp%3a%2f%2ftracker.kamigami.org%3a2710%2fannounce&tr=udp%3a%2f%2ftracker.grepler.com%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.filetracker.pl%3a8089%2fannounce

ymarcus93 commented 7 years ago

Definitely not 17GB or 21GB...I'm getting 39GB and still downloading for windsat. In my logs, it seems like there is a .snapshot folder in many of the subfolders. I think your original estimate didn't take these hidden folders into account. My wget --mirror is downloading these anyways, although I don't know if .snapshot has duplicates.

UPDATE: Yah. .snapshot has duplicates, so no need to mirror that.

ymarcus93 commented 7 years ago

Seems like this data is updated daily. Same exact data is also available at: http://data.remss.com/windsat/.

bkirkbri commented 7 years ago

@markuslaker posting magnet links is helpful. We hope to aggregate the comments in these issues into a searchable/browsable index soon.

@axlecrusher @erikfriesen Are your mirrors public?