search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Dataset at ftp:/ftp.star.nesdis.noaa.gov/pub/sod #166

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

ftp:/ftp.star.nesdis.noaa.gov/pub/sod.

Suggested in a large email containing many urls

Artoria2e5 commented 7 years ago

Weird one. wget doesn't work, but both Chrome and Windows Explorer work fine exploring.

Artoria2e5 commented 7 years ago

By looking at the .listing fetched, it seems that sod is a symlink to socd. Should I try ftp.star.nesdis.noaa.gov/pub/socd with wget?

drwxr-sr-x  10 root     root          146 Apr 27  2016 .
drwxr-xr-x   3 root     root         4096 Jun 15  2015 ..
drwxr-sr-x   9 root     30004          91 Sep 25  2014 corp
drwxrwsr-x  36 root     30002        4096 Dec 19 20:04 smcd
drwxrwsr-x  10 root     30005         132 Jan 13 13:02 socd
drwxrwsr-x   4 root     30005          39 Jan  6 16:37 socd1
drwxrwsr-x   4 root     30005          46 Aug  1 15:58 socd2
drwxrwsr-x   3 root     30005          23 Jan  6 16:34 socd3
lrwxrwxrwx   1 root     root            4 Jan  2  2015 sod -> socd
drwxr-sr-x  12 root     30003        4096 Feb 14  2014 star
drwxrwsr-x   8 root     3010         4096 Dec  6 14:31 tst

Wow, that works. Downloading, not sure about how long my battery will last.

Artoria2e5 commented 7 years ago

A very partial download for socd/{mecb,orad}, intended as a headstart : https://drive.google.com/open?id=0Bxaa9Ck2sP8fcDNpR0JIZ0pmQUU. Requires lrzip (x64) for decompression.

wsh commented 7 years ago

Attempting to mirror entire directory now.

Temporarily suspended because utility of data unclear.

donbright commented 7 years ago

Rough size estimate: 58T using lftp du -h

JeremiahCurtis commented 7 years ago

Could anyone break this down by folder size and perhaps begin different issues?

Much appreciated

Artoria2e5 commented 7 years ago

tar+lrzip archive (~11GB) for lsa/cs2igdr (~35 GB; CryoSat Interim Geophysical Data Record; many small files): https://drive.google.com/open?id=0B2lJgc3uTwAFcFU5WUJQTWZReUU

Signature:

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAABCgAGBQJYiu7QAAoJEFXHLSNoWrnkKtoH/iAW3vPF6CaOzoBJy8ViJYre
6lIcCb9UXj1PP0XdDlhkm6lejyiy3ckAqtgVn9H/76HrKno1JEaJvAVWDpczs8bE
WaUkDUlo3V2QousyBOXhWG8zaF6CToH+MxnVayEkR99ybX1pE8LzwTCS/qfSg98Z
qMDKLc0XSLU47Hl5hQwsDLuEV3HlCY2vnx5zmdcz1f6jeyjB/vy0zaevR07ygxER
tknP3Q3aXmibyN2sGe2TRLuBSxSzMRlL0tQU6C+WCsfnLI29rfzCmRHM3nXREXyj
3uZ1Bg6Ba4rBYP/2mrc07sHb7PEv++gZiRPyvh+GXKlzIaW2ah9kjjLiyW1tiwU=
=EyQn
-----END PGP SIGNATURE-----
sarunasb commented 7 years ago

Individual folder sizes:

lftp ftp.star.nesdis.noaa.gov:/pub/socd>
drwx------   2 root     30005           6 Oct 27  2009 lost+found
drwxrwxr-x  18 3688     3000         4096 Mar  9 19:32 lsa               425 GB
drwxrwsr-x  21 root     30005        4096 Dec 22 19:38 mecb               23 TB
drwxrwsr-x  36 root     30005        4096 Feb 24  2016 orad              346 GB
drwxrwsr-x  16 root     30005        4096 Dec  7 00:07 osb                 6 TB
drwxr-sr-x   2 3514     30005        4096 Jan 13 13:21 sss                31 MB
drwxrwsr-x   6 root     30005          85 Feb 29  2016 sst                18 TB
drwxr-sr-x  17 3728     30005        4096 Sep  8  2016 xzhu              282 GB
sarunasb commented 7 years ago

Downloading ftp.star.nesdis.noaa.gov:/pub/socd/sst to geom.dartmouth.edu.

sarunasb commented 7 years ago

ftp.star.nesdis.noaa.gov:/pub/socd/sst mirror, checksums on geom.dartmouth.edu.