search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Oak Ridge National Laboratory DAAC #42

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Name: Oak Ridge National Laboratory DAAC Organization: NASA-ESDIS Description URL: https://daac.ornl.gov/ Download URL: https://daac.ornl.gov/get_data.shtml File Types: Size: Status:

ghost commented 7 years ago

This is Issue #26 of The Azimuth Backup Project. We have made a start on it, but the server seems not to hold up it's end, or throttles. We are using:

wget -N -c --dns-timeout=10 --connect-timeout=300 --read-timeout=120 --wait=5 --mirror -e robots=off --random-wait --page-requisites --retry-connrefused --prefer-family=IPv4 --tries=40 --timestamping=on --recursive --level=8 --no-remove-listing --follow-ftp -nv --mirror --append-output=daac-ornl-gov-get-data.log --no-check-certificate https://daac.ornl.gov/

And we get things like:

2017-01-20 15:27:42 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1146 [37050] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1146" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:27:53 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1080 [38162] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1080" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:01 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1166 [35130] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1166" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:11 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=817 [35196] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=817" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:24 URL:http://daac.ornl.gov/cgi-bin/dataset_lister_new.pl?p=34 [53907] -> "daac.ornl.gov/cgi-bin/dataset_lister_new.pl@p=34" [1]
WARNING: cannot verify urs.earthdata.nasa.gov's certificate, issued by `/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M':
  Unable to locally verify the issuer's authority.
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:34 URL:https://urs.earthdata.nasa.gov/oauth/authorize?client_id=QyeRbBJg8YuY_WBh-KBztA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=aHR0cDovL2RhYWMub3JubC5nb3YvZGFhY2RhdGEvYWJvdmUv [8950] -> "urs.earthdata.nasa.gov/oauth/authorize@client_id=QyeRbBJg8YuY_WBh-KBztA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=aHR0cDovL2RhYWMub3JubC5nb3YvZGFhY2RhdGEvYWJvdmUv" [1]
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1307:
2017-01-20 15:29:47 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1362:
2017-01-20 15:30:28 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1367:
2017-01-20 15:30:38 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1324:
2017-01-20 15:31:10 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1355:
2017-01-20 15:31:32 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1353:
2017-01-20 15:31:44 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1354:
2017-01-20 15:32:05 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1356:
2017-01-20 15:32:34 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1360:
2017-01-20 15:32:49 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1333:
2017-01-20 15:33:13 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1361:
2017-01-20 15:33:27 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1323:
2017-01-20 15:33:40 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1265:
2017-01-20 15:33:47 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1265:
2017-01-20 15:34:05 ERROR 404: Not Found.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.

A restart of the download seems to begin okay. I have attached a zipped copy of a log from this. daac-ornl-older.log.gz

mejackreed commented 7 years ago

Is there a size estimate here?

ghost commented 7 years ago

Our best grab is 285 GB:

but I know its incomplete. In the process of moving it and picking it up again, unless someone else can.

ghost commented 7 years ago

I'm working on seeing if there is some way we can put a tarball of the dataset some place on a server of The Azimuth Project in a way it could be shared. This is definitely a partial, but wanted to help out if someone else wanted to pick up and continue.

mejackreed commented 7 years ago

A have a wget job running, but I think bandwidth on their ftp may be limited. I'm struggling to get even close to that amount of data so far.

JeremiahCurtis commented 7 years ago

working on the CARVE and CMS databases, but the latter is about 460 GB and is taking a long time

JeremiahCurtis commented 7 years ago

If somebody with a faster connection can pick up CMS data, go for it....my connection to ORNL is abjectly poor

JeremiahCurtis commented 7 years ago

Any idea where we stand on this? If certain EPA pages referring to climate change are being removed, I can't imagine ORNL data being safe much longer. Not trying to cry wolf, just saying.....

My CARVE and CMS downloads died on download manager and so I am reordering them from Earthdata

Also, are there any FTP directories for ORNL data?

ghost commented 7 years ago

https://bitbucket.org/azimuth-backup/azimuth-inventory/wiki/Our%20progress

On Fri, Feb 3, 2017, at 14:55, JeremiahCurtis wrote:

Any idea where we stand on this? If certain EPA pages referring to climate change are being removed, I can't imagine ORNL data being safe much longer. Not trying to cry wolf, just saying..... My CARVE and CMS downloads died on download manager and so I am reordering them from Earthdata Also, are there any FTP directories for ORNL data?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub[1], or mute the thread[2].

Links:

  1. https://github.com/climate-mirror/datasets/issues/42#issuecomment-277346257
  2. https://github.com/notifications/unsubscribe-auth/AD3HBw7LSYg0qV7iiqnOz16S05L5JXBEks5rY4ZFgaJpZM4LhVBk
JeremiahCurtis commented 7 years ago

new to azimuth, so I apologize for what might be a stupid question; I'm assuming "resolved" means that at least one copy of all ORNL data, including CMS, CARVE, Daymet, etc. exists?