search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp:/space.mit.edu/pub/plasma #219

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/space.mit.edu/pub/plasma.

Suggested in a large email containing many urls

siennathesane commented 7 years ago

@nickrsan do we need to mirror MIT data? I think a university would be unaffected, unless they are stopping all climate research.

nickrsan commented 7 years ago

Agreed - low priority for sure.

Colt8027 commented 7 years ago

Grabbed the full mirror while doing another 2. This one has a few weird caveats. Size: 22GB Size on disk: 22.1GB Contains: 47,753 Files, 992 Folders

Compressed 7z: 7.84GB

gsg folder failed
jtest empty
lost+found empty
www folder failed
21 failed downloaded files "Permission Denied" http://pastebin.com/8hk3qJeK

How would you like me to mirror it? My box I'm using is 1Gbe Up/Down so upload isn't a issue.

siennathesane commented 7 years ago

Torrents are good, some people are using S3/GCE/GDrive, it's completely up to you.

Colt8027 commented 7 years ago

@mxplusb What the general idea around here for mirroring it, Since its so many files, I believe compressed format would be much better for torrent if I go that route.

siennathesane commented 7 years ago

Right now there is no official method of mirroring, some people are compressing the data, some are not. My advise is to use whichever format works best for your situation, compressing the data is completely acceptable. :)

Colt8027 commented 7 years ago

@mxplusb Good deal! I created the torrent, where are we posting .torrent/magnet links? or are we just submitting to climatemirror.org/ ?