search

climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

NOAA NCIE Datasets #274

Open siennathesane opened 7 years ago

siennathesane commented 7 years ago

Here are all the NOAA National Center for Environmental Information (NCIE) datasets.

I have no idea how much data this is, but it's going to be a lot.

https://www.ncei.noaa.gov/data/

siennathesane commented 7 years ago

I'm downloading the Ocean Heat Flux datasets first, then I'll review the priorities of the rest of it.

blueacid commented 7 years ago

I'm checking roughly how large each folder is.. so far this is what I've got (I'll keep editing to update this comment)

lftp www.ncei.noaa.gov:/data> du -h . 1.9G ./amsu-a-1c-bt-grid/access/2016/AQUA/C20160822 27G ./amsu-a-1c-bt-grid/access/2016/AQUA/C20160824 160M ./amsu-a-1c-bt-grid/access/2016/AQUA/C20160906 29G ./amsu-a-1c-bt-grid/access/2016/AQUA 2.1G ./amsu-a-1c-bt-grid/access/2016/M02/C20160822 3.1G ./amsu-a-1c-bt-grid/access/2016/M02/C20160824 0 ./amsu-a-1c-bt-grid/access/2016/M02/C20160825 0 ./amsu-a-1c-bt-grid/access/2016/M02/C20160906 5.1G ./amsu-a-1c-bt-grid/access/2016/M02 0 ./amsu-a-1c-bt-grid/access/2016/N15/C20160822 0 ./amsu-a-1c-bt-grid/access/2016/N15/C20160825 0 ./amsu-a-1c-bt-grid/access/2016/N15/C20160906 0 ./amsu-a-1c-bt-grid/access/2016/N15 0 ./amsu-a-1c-bt-grid/access/2016/N16 0 ./amsu-a-1c-bt-grid/access/2016/N17 0 ./amsu-a-1c-bt-grid/access/2016/N18 34G ./amsu-a-1c-bt-grid/access/2016 34G ./amsu-a-1c-bt-grid/access 34G ./amsu-a-1c-bt-grid 1.9G ./amsu-a-brightness-temperature-noaa/access/2016/AQUA/C20160822 27G ./amsu-a-brightness-temperature-noaa/access/2016/AQUA/C20160824 160M ./amsu-a-brightness-temperature-noaa/access/2016/AQUA/C20160906 29G ./amsu-a-brightness-temperature-noaa/access/2016/AQUA 2.1G ./amsu-a-brightness-temperature-noaa/access/2016/M02/C20160822 17G ./amsu-a-brightness-temperature-noaa/access/2016/M02/C20160824 1.2G ./amsu-a-brightness-temperature-noaa/access/2016/M02/C20160825 161M ./amsu-a-brightness-temperature-noaa/access/2016/M02/C20160906 20G ./amsu-a-brightness-temperature-noaa/access/2016/M02 1.9G ./amsu-a-brightness-temperature-noaa/access/2016/N15/C20160822 32G ./amsu-a-brightness-temperature-noaa/access/2016/N15/C20160825 161M ./amsu-a-brightness-temperature-noaa/access/2016/N15/C20160906 34G ./amsu-a-brightness-temperature-noaa/access/2016/N15 8.3G ./amsu-a-brightness-temperature-noaa/access/2016/N16/C20160825 8.3G ./amsu-a-brightness-temperature-noaa/access/2016/N16 3.0G ./amsu-a-brightness-temperature-noaa/access/2016/N17/C20160825 3.0G ./amsu-a-brightness-temperature-noaa/access/2016/N17 2.3G ./amsu-a-brightness-temperature-noaa/access/2016/N18/C20160822

siennathesane commented 7 years ago

I have the Ocean Heat Flux dataset.

FINISHED --2017-01-27 12:22:40--
Total wall clock time: 2d 1h 17m 59s
Downloaded: 10429 files, 269G in 1d 23h 9m 9s (1.62 MB/s)
blueacid commented 7 years ago

My lftp du command died.. but that's a hint of just how massive this resource is: few hundred gig for just the AMSU data that I looked at so far. Since your ocean-heat-fluxes folder was 269GB, and there's about 50-60 folders... well, if they're all 200GB+ that's loads to grab. More attention on this one I suggest? ...having said that, snow-cover-extent weighs in at 24MB. Huh.

blueacid commented 7 years ago

Grabbing sea-ice-concentration; less talk, more downloading :)

planetor commented 7 years ago

I'm downloading this directory to my server: https://www.ncei.noaa.gov/data/

I downloaded 331G! Still running...

planetor commented 7 years ago

OMG! Can anyone mirror this directory? It's too big! There are more than 800GB: https://www.ncei.noaa.gov/data/igra/v2beta/

wantonwonton commented 7 years ago

Okay, I've started downloading https://www.ncei.noaa.gov/data/igra/v2beta/. I have enough space, though my download speed maxes out at around 4MB/s, so this will take 2+ days.

wantonwonton commented 7 years ago

So, https://www.ncei.noaa.gov/data/igra/v2beta/ seems to be over 2.5TB. I did a little digging, and found the current IGRA data at ftp://ftp.ncdc.noaa.gov/pub/data/igra/ and https://www1.ncdc.noaa.gov/pub/data/igra/. The igra2-readme.txt file talks about the v2 data.

According to https://geosci.uchicago.edu/~rtp1/PrinciplesPlanetaryClimate/Data/WorkbookDatasets/Chapter2Data/IGRA_sondes/readmeIGRA.txt, the data-por files are updated daily with the latest day's data added: "data-por Period of record data files (base version through 2006 + daily updates; one file per station)".

The ftp://ftp.ncdc.noaa.gov/pub/data/igra/data/data-por/ directory seems to be about 19GB. In the v2beta directory, the IGRA_v2.0data-por*.tar files also seem to be about that same size and since their filenames have dates in them, it looks like they are probably snapshots of the IGRA v2 data that were taken while it was being beta tested. Each daily snapshot was probably intended to be a superset of the prior day's snapshot (excluding any glitches).

So, it seems like the v2beta data may not be that important (due to most of it likely being redundant) and I've stopped downloading the full v2beta directory (after getting about 400GB). I am grabbing the latest instance of each series of v2beta files, and will download ftp://ftp.ncdc.noaa.gov/pub/data/igra/.

planetor commented 7 years ago

@wantonwonton

Yes your are right it seems to be not important. I've found a readme.txt file about it:

https://www1.ncdc.noaa.gov/pub/data/igra/status.txt

**September 11, 2014 Addition

A beta version of IGRA v2 is now available, including sounding data, monthly means, derived parameters, station history information, and basic format documentation. IGRA 2beta is available in the FTP directory http://www1.ncdc.noaa.gov/pub/data/igra/v2beta/ or at ftp.ncdc.noaa.gov under pub/data/igra/v2beta. All other files in http://www1.ncdc.noaa.gov/pub/data/igra/ as well as the webpages at http://www.ncdc.noaa.gov/oa/climate/igra/ will continue to reflect

IGRA 1 until IGRA 2 has completed NCDC's internal review process and been approved as NCDC's official replacement of IGRA 1. For more information on IGRA 2, see http://www1.ncdc.noaa.gov/pub/data/igra/v2beta/igra2-readme.txt **

We should check big files whether it's necessary to download (readme files or websites). But generally, if you have an archive of all historic data as a tar or tar.gz file please download this single one instead of every single station file.

My download is still running...

bkirkbri commented 7 years ago

I've got a mirror of www.ncei.noaa.gov/data/marine/icoads3.0 though it's only a small dataset (~3gb). It is available at http://climate-mirror.s3.ca-central-1.amazonaws.com/index.html

Thanks to @mxplusb @blueacid @planetor @wantonwonton

Please update as mirrors complete.

wantonwonton commented 7 years ago

I've downloaded the latest file of each type/series from the https://www.ncei.noaa.gov/data/igra/v2beta/ directory (plus a bunch of the earlier IGRA_v2.0data-por*.tar files that I'm not listing here):

-rw-rw-r-- 1 root root 19712624640 Oct 27 15:14 IGRA_v2.0_data-por_s19050404_e20161026_c20161026.tar -rw-rw-r-- 1 root root 524288000 Oct 27 15:21 IGRA_v2.0_data-y2d_s20160101_e20161026_c20161026.tar -rw-rw-r-- 1 root root 20480 Aug 16 13:56 IGRA_v2.0_doc-data_c20160815.tar -rw-rw-r-- 1 root root 10240 Aug 17 13:49 IGRA_v2.0_doc-derived_c20160816.tar -rw-rw-r-- 1 root root 296960 Oct 30 13:49 IGRA_v2.0_doc-general_c20161029.tar -rw-rw-r-- 1 root root 27832320 Sep 9 15:33 IGRA_v2.0_doc-history_c20160826.tar -rw-rw-r-- 1 root root 10240 Sep 9 15:11 IGRA_v2.0_doc-monthly_c20160906.tar -rw-rw-r-- 1 root root 727040 Sep 9 15:11 IGRA_v2.0_monthly-upd_s201608_e201608_c20160906.tar

MD5 sums:

a6bb29cf614c9232e62b732ddfb6250e IGRA_v2.0_data-por_s19050404_e20161026_c20161026.tar 1dde9a2f62ec5215d73a3ce2ab71990e IGRA_v2.0_data-y2d_s20160101_e20161026_c20161026.tar 450e9694418c2b06be0f033ba69b140c IGRA_v2.0_doc-data_c20160815.tar 946358acec2d88317997939f447c4702 IGRA_v2.0_doc-derived_c20160816.tar 391678b29b11334937dfc2546ebf85a0 IGRA_v2.0_doc-general_c20161029.tar 06d44d8dbc54c488ad063b35a21e045e IGRA_v2.0_doc-history_c20160826.tar 118fadfa605f1a5ee1160c66205cb1e3 IGRA_v2.0_doc-monthly_c20160906.tar bf204492c89cbe4acf3788b86b5c5179 IGRA_v2.0_monthly-upd_s201608_e201608_c20160906.tar

I'm still downloading the current IGRA data from https://www1.ncdc.noaa.gov/pub/data/igra/. Note that since each of the data files is updated daily, my snapshot likey won't be a clean one from a single day.

planetor commented 7 years ago

My current state is:

1 = done 2 = please download, because my webspace has reached its limits 3 = download of this directory is still running 4 = directory was empty

~ 950G total

blueacid commented 7 years ago

sea-ice-concentration has finished; 71.5GB.

I'll sort out links and suchlike soon

JeremiahCurtis commented 7 years ago

So far, I have the following: climat-bufr outgoing-longwave-radiation-daily outgoing-longwave-radiation-monthly precipitation-persiann-cdr snow-cover-extent snow-cover-extent-northern-hemisphere snowstorm spectral-solar-irradiance

Not sure how to double-check these, however.....I'm a newbie at mirroring so I checked them manually on IDM and all files showed complete. Those that did not, I grabbed with downthemall

update: avhrr-aerosol-optical-thickness is almost finished and crawling to the finish line

JeremiahCurtis commented 7 years ago

working on sea surface temperature optimum interpolation and whoi; if there is freeware better than internet download manager, it would help, as IDM keeps throwing error messages especially when running four or more grab projects simultaneously

wantonwonton commented 7 years ago

My download of https://www1.ncdc.noaa.gov/pub/data/igra/ finished. Note that some of the files were updated on different days.

JeremiahCurtis commented 7 years ago

Not sure that I have the space or speed for the following (I have not started any of these btw): amsu-a-1c-bt-grid amsu-a-brightness-temperature-noaa avhrr-aerosol-optical-thickness avhrr-land-leaf-area-index-and-fapar avhrr-land-normalized-difference-vegetation-index avhrr-land-surface-reflectance avhrr-polar-pathfinder avhrr-polar-pathfinder-extended avhrr-reflectance-cloud-properties-patmos-extended

planetor commented 7 years ago

@JeremiahCurtis My rough calculations are:

amsu-a-brightness-temperature-noaa/access/2016 ~148.5GB ~ 49440 files amsu-a-1c-bt-grid/access/2016/M02/C20160822 ~2200MB amsu-a-1c-bt-grid/access/2016/M02/C20160824/ ~3200MB
avhrr-land-surface-reflectance/ ~1500GB
avhrr-land-normalized-difference-vegetation-index/ ~750GB
avhrr-land-leaf-area-index-and-fapar/ ~75GB
avhrr-aerosol-optical-thickness/access/daily/ ~255.5GB
avhrr-aerosol-optical-thickness/access/monthly/ ~8.5GB
avhrr-polar-pathfinder-extended/access/ ~766.5GB
avhrr-reflectance-cloud-properties-patmos-extended/access/ ~15000GB i've seen up to 3000 files per directory

JeremiahCurtis commented 7 years ago

I'm grabbing avhrr-aerosol-optical-thickness/access/monthly/ on Internet Download Manager; will attempt avhrr-land-leaf-area-index-and-fapar/ when that is done

Beyond that, I will probably bow out of the avhrr and amsu directories because of storage shortage

blueacid commented 7 years ago

http://climatedata.project.zone/mirror/www.ncei.noaa.gov/data/sea-ice-concentration/ Sea Ice Concentration is done

bkirkbri commented 7 years ago

Can someone volunteer to keep track of the many efforts going on in this issue? This issue could either be broken into an issue per directory or tracked in here. As long as there is some way to be sure of the status. Thanks!

JeremiahCurtis commented 7 years ago

avhrr-aerosol-optical-thickness/access/monthly/ finished working on sea-surface-temperature-optimum-interpolation and sea-surface-temperature-whoi, but they keep hanging on internet download manager

idm has been stuck on https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/access/avhrr-only/198703/avhrr-only-v2.19870305.nc for several hours (I opened and reopened IDM several times), and several files before that didn't completely finish

FWIW, I'm a neophyte at this and can't seem to get IDM to cooperate

planetor commented 7 years ago

Done! My disk is full! Total size of backup: 1.3T

#########################################################

Dataset URL: www.ncei.noaa.gov/data/climat-bufr/ Name: ClimatBufr 2016 Metadata: http://en.wikipedia.org/wiki/BUFR Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/climat-bufr/ Size: 144K Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/climdiv/ Name: ClimDiv - DIVISIONAL TEMPERATURE-PRECIPITATION-DROUGHT - JUNE 2014 Metadata: http://data.noaa.gov/harvest/object/60451fa9-7af9-418b-a356-12e99db30dbf/html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/climdiv/ Size: 164M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ghcnm/v3/ Name: GHCN version 3 Land Temperature Anomalies from GHCN-M V3 Metadata: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/README Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ghcnm/v3/ Size: 1.6G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ghcnm/v4/ Name: GLOBAL HISTORICAL CLIMATOLOGY NETWORK - MONTHLY (GHCN-M) Version 4 Metadata: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v4/beta/readme.txt Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ghcnm/v4/ Size: 77M Responsible Agency: NOAA

Notes: Look at BETA RELEASE ANNOUNCEMENT on ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v4/beta/readme.txt

#########################################################

Dataset URL: www.ncei.noaa.gov/data/gpsmet/ Name: Ground-Based Global Positioning System (GPS) Meteorology Integrated Precipitable Water Vapor (IPW) Metadata: http://data.noaa.gov/harvest/object/232cbb81-3cd4-4fd7-92ae-93e2cc2cb5b6/html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/gpsmet/ Size: 18G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/gsom/ Name: Ground-Based Global Positioning System (GPS) Meteorology Integrated Precipitable Water Vapor (IPW) Metadata: http://data.noaa.gov/harvest/object/de3086a9-2aee-4b97-85cc-c9a349c3c42d/html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/gsom/ Size: 5.8G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/marine/icoads3.0/ Name: International Comprehensive Ocean-Atmosphere Data Set (ICOADS) V3 Metadata: http://icoads.noaa.gov/products.html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/marine/icoads3.0/ Size: 33G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/mean-layer-temperature-noaa/ Name: Mean Layer Temperature - NOAA Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/mean-layer-temperature-noaa Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/mean-layer-temperature-noaa/ Size: 79M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/mean-layer-temperature-rss Name: Mean Layer Temperature - RSS Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/mean-layer-temperature-rss Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/mean-layer-temperature-rss Size: 311M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/mean-layer-temperature-uah Name: Mean Layer Temperature - UAH Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/mean-layer-temperature-uah Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/mean-layer-temperature-uah Size: 49M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/mean-layer-temperature-ucar-lower-stratosphere/ Name: Mean Layer Temperature - UCAR (Lower Stratosphere) Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/mean-layer-temperature-ucar-lower-stratosphere Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/mean-layer-temperature-ucar-lower-stratosphere/ Size: 35M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/mean-layer-temperature-ucar-upper-trop-lower-strat/ Name: Mean Layer Temperature - UCAR (Upper Trop & Lower Strat) Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/mean-layer-temperature-ucar-upper-trop-lower-strat Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/mean-layer-temperature-ucar-upper-trop-lower-strat/ Size: 29M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/msu-brightness-temperature-noaa/ Name: MSU Brightness Temperature - NOAA Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/msu-brightness-temperature-noaa Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/msu-brightness-temperature-noaa/ Size: 76G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ssmis-brightness-temperature-rss/ Name: SSMI(S) Brightness Temperature - RSS Metadata: http://www.ncdc.noaa.gov/cdr/fundamental/ssmis-brightness-temperature-rss Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ssmis-brightness-temperature-rss/ Size: 267G (from 1987 - 1992) ~ 50G/directory Responsible Agency: NOAA

Notes: Backup only from 1987 to 1994! (for 1995 - 2016 no backup)

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ocean-near-surface-atmospheric-properties/ Name: Ocean Near-surface Atmospheric Properties Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/ocean-near-surface-atmospheric-properties Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ocean-near-surface-atmospheric-properties/ Size: 392G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/outgoing-longwave-radiation-daily/ Name: Outgoing Longwave Radiation - Daily Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/outgoing-longwave-radiation-daily Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/outgoing-longwave-radiation-daily/ Size: 3.2G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/outgoing-longwave-radiation-monthly/ Name: Outgoing Longwave Radiation - Monthly Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/outgoing-longwave-radiation-monthly Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/outgoing-longwave-radiation-monthly/ Size: 36M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ozone-concentration-esrl/ Name: Ozone Concentration ESRL Metadata: http://www.esrl.noaa.gov/gmd/about/ozone.html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ozone-concentration-esrl/ Size: 67M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/precipitation-persiann Name: Precipitation - PERSIANN-CDR Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/precipitation-persiann-cdr Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/precipitation-persiann Size: 13G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/precipitation-persiann-cdr Name: Precipitation - PERSIANN-CDR Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/precipitation-persiann-cdr Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/precipitation-persiann-cdr Size: 184M Responsible Agency: NOAA

Notes: Only 2016 - 2017

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ratpac/ Name: Radiosonde Atmospheric Temperature Products for Assessing Climate (RATPAC) Metadata: http://www.ncdc.noaa.gov/data-access/weather-balloon/radiosonde-atmospheric-temperature-products-accessing-climate Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ratpac/ Size: 23M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/snow-cover-extent/ Name: Sea Ice and Snow Cover Extent Metadata: http://www.ncdc.noaa.gov/snow-and-ice/extent/ Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/snow-cover-extent/ Size: 24M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/snow-cover-extent-northern-hemisphere/ Name: Sea Ice and Snow Cover Extent Metadata: http://data.noaa.gov/harvest/object/07a7c384-d17a-4bbd-95bf-2b6f5a860a33/html Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/snow-cover-extent-northern-hemisphere/ Size: 48M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/snowstorm/ Name: Regional Snowfall Index (RSI) and the Northeast Snowfall Impact Scale (NESIS) Metadata: www.ncei.noaa.gov/data/snowstorm/regional-snowfall-index_readme_c20130529.txt Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/snowstorm/ Size: 60M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/ssw/ Name: SSWC: Sudden Stratospheric Warming Compendium data set Metadata: http://www.esrl.noaa.gov/csd/groups/csd8/sswcompendium/ Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/ssw/ Size: 303G Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/total-solar-irradiance Name: Total Solar Irradiance Metadata: http://www.ncdc.noaa.gov/cdr/atmospheric/total-solar-irradiance Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/total-solar-irradiance Size: 27M Responsible Agency: NOAA

#########################################################

Dataset URL: www.ncei.noaa.gov/data/uscrn/ Name: U.S. Climate Reference Network (USCRN) Metadata: http://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/us-climate-reference-network-uscrn Mirror URL: http://h2490363.stratoserver.net/climatemirror/www.ncei.noaa.gov/data/uscrn/ Size: 7.0G Responsible Agency: NOAA

JeremiahCurtis commented 7 years ago

still working on sea-surface-temperature-optimum-interpolation and amsu-a-1c-bt-grid, both via internet download manager.... No problems so far except for lack of speed

so far, on amsu-a-1c-bt-grid, I'm at 7.92 GB (8,510,936,894 bytes) and counting......anyone know the complete size?

update: finished amsu-a-1c-bt-grid 9.27 GB (9,954,436,827 bytes)

bkirkbri commented 7 years ago

@planetor Thanks for the detailed breakdown of mirrors! Can you calculate hashdeep -rl output for each and post links to the hashdeep files? That would be very helpful.

planetor commented 7 years ago

@bkirkbri I did that 12 days ago: Note: Each md5 list was sorted with "sort -k2"

All hashes in one file: http://h2490363.stratoserver.net/climatemirror/md5sum_files_sorted_list.csv

Separated by directory / post: http://h2490363.stratoserver.net/climatemirror/md5/

gabefair commented 7 years ago

I have created a new ticket for the /ghcn data #331

JeremiahCurtis commented 7 years ago

Finished amsu-a-1c-bt-grid Finished https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/access/amsr-avhrr/ Re-grabbing https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/access/avhrr-only/ ......should be finished later today

JeremiahCurtis commented 7 years ago

As far as I can tell, from reviewing the many issues here, this would be where we stand on the NCEI folders:

LOCAL amsu-a-1c-bt-grid/ OPEN amsu-a-brightness-temperature-noaa/ OPEN avhrr-aerosol-optical-thickness/ LOCAL avhrr-land-leaf-area-index-and-fapar/ OPEN avhrr-land-normalized-difference-vegetation-index/ OPEN avhrr-land-surface-reflectance/ OPEN avhrr-polar-pathfinder-extended/ OPEN avhrr-polar-pathfinder/ OPEN avhrr-reflectance-cloud-properties-patmos-extended/ DONE climat-bufr/ DONE climdiv/ DONE climgrid/ LOCAL epa/ DONE (SEE ISSUE #313) geostationary-ir-channel-brightness-temperature-gridsat-b1/ EMPTY ghcnd/ DONE ghcnm/ EMPTY global-hourly/ EMPTY global-marine/ OPEN gpcp/ DONE gpsmet/ LOCAL gridmasks/ DONE gsom/ LOCAL gsoy/ DONE hirs-brightness-temperature/ UNSURE; PLEASE ADVISE igra/ EMPTY isd/ ICOADS 3.0 DONE; NEED OTHER SUBFOLDERS marine/ DONE mean-layer-temperature-noaa/ DONE mean-layer-temperature-rss/ DONE mean-layer-temperature-uah/ DONE mean-layer-temperature-ucar-lower-stratosphere/ DONE mean-layer-temperature-ucar-upper-trop-lower-strat/ DONE msu-brightness-temperature-noaa/ DONE ocean-heat-fluxes/ DONE ocean-near-surface-atmospheric-properties/ DONE outgoing-longwave-radiation-daily/ DONE outgoing-longwave-radiation-monthly/ DONE ozone-concentration-esrl/ WORKING precip-15/ OPEN precip-hly/ DONE precipitation-persiann/ DONE ratpac/ DONE sea-ice-concentration/ LOCAL sea-surface-temperature-optimum-interpolation/ REVIEWING sea-surface-temperature-whoi/ DONE snow-cover-extent-northern-hemisphere/ DONE snow-cover-extent/ DONE snowstorm/ LOCAL spectral-solar-irradiance/ WORKING ON /BASE; NEED OTHER SUBFOLDERS ssmis-brightness-temperature-csu/ NEED 1995 ONWARD ssmis-brightness-temperature-rss/ DONE ssw/ DONE total-solar-irradiance/ DONE uscrn/

LOCAL= I have finished downloading the folders in question to an external hard drive, and will make it publicly available ASAP (also available upon request in the meantime) DONE=indicated as finished by others, such as, among others, @mxplusb and @planetor indicated above on this or other issues WORKING=I have the folders in question under download or on queue REVIEWING=I am reviewing my copy of this dataset due to the fact that the download was interrupted at several points EMPTY=nothing in directory

I will probably open a few issues for the larger folders that have not yet been addressed. Judging from what is here on github, we probably have barely scratched the surface of this; these folders alone add up to about 25TB:

amsu-a-brightness-temperature-noaa/ avhrr-aerosol-optical-thickness/ avhrr-land-normalized-difference-vegetation-index/ avhrr-land-surface-reflectance/ avhrr-polar-pathfinder-extended/ avhrr-polar-pathfinder/ avhrr-reflectance-cloud-properties-patmos-extended/