search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

Carbon Dioxide Information Analysis Center #7

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Name: Carbon Dioxide Information Analysis Center Organization: DOE/ORNL Description URL: http://web.archive.org/web/20161230211915/http://cdiac.ornl.gov/ Download URL: File Types: Size: Mirror URL: http://web.archive.org/web/20170304212711/http://cdiac.ornl.gov/data_catalog.html

gabefair commented 7 years ago

Copy of notice on site: NOTICE (August 2016): CDIAC as currently configured and hosted by ORNL will cease operations on September 30, 2017. Data will continue to be available through this portal until that time. Data transition plans are being developed with DOE to ensure preservation and availability beyond 2017.

gabefair commented 7 years ago

@nickrsan correct me if I'm wrong, but I don't think the Internet Archive has a copy of the data, just the project site. I will now remove the IA as a mirror

rustyguts commented 7 years ago

I have completed a compressed version of this dataset. It was 70GB Mirrors

gabefair commented 7 years ago

Hi @RustyGuts thank you for downloading this. How did you download this data? I was unsure of how to approach this one since the datasets are located here: http://cdiac.ornl.gov/data_catalog.html# and there are lots of children pages to download.

rustyguts commented 7 years ago

@gabefair I just ran HTTrack, it did a great job at scooping up multiple levels of data. It's worth noting that while I verified most of the downloaded data, you can't be 100% positive with web scoopers like that.

If someone else could run HTTrack or something similar and verify 70GB worth of data then we can be pretty positive. I'm also working on getting decompressed shares of this data available on ACD since downloading 70gb just to check data integrity is a bother.

CorentinB commented 7 years ago

I have a mirror of this here : https://archives.corentinb.me/content/climate-mirror-datasets/7.html