EPA Air Quality Data archives

rsargent commented 7 years ago

New Dataset

Please fill out the New Issue form so we can easily organise the issues and put a priority on certain data. The title should reflect the dataset you want us to save. In the body, please include the following information.

Agency: EPA
Agency Division: Air Quality
Data Type: Periodic measures of various air pollutants at various monitoring sites
Data Format: zipped textual tables
FTP/HTTP URL: http://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/download_files.html

By adding the government agency and the division which collects the data, it will make it much easier to catalogue and prioritise!

Example:

Agency: NOAA
Agency Division: NCEI
Data Type: Oceanographic
Data Format: Unknown (it's okay if you don't know)
FTP/HTTP URL: https://www.ncei.noaa.gov/data/ocean-near-surface-atmospheric-properties/

JeremiahCurtis commented 7 years ago

Finished....965 zip files 12.7 GB (13,661,205,194 bytes)

HostileGranola commented 7 years ago

I'm currently downloading this using aria2c -i list.txt -d aqsdr1.epa.gov/aqsweb/aqstmp/airdata/ -x 10 -c. The list is a plaintext version of this file stripped down to the first column, with "http://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/" appended to the start of each line.

I will post hashes from hashdeep -erl and file sizes from du -b --max-depth=1 --human-readable once complete.

HostileGranola commented 7 years ago

I have a copy of this dataset as of 2017-03-28 02:40:00 UTC

Hash results are here

Size results are here

HostileGranola commented 7 years ago

@JeremiahCurtis I notice our downloads are different sizes, is it possible for you to compare your results with mine? I can make my list file available if that would be of use to you.

Juerd commented 7 years ago

http://[2a03:b0c0:2:d0::1dae:1001]/EPA_Air_Quality_Data/ http://188.166.4.6/EPA_Air_Quality_Data/

Will stay online until the 1 TB transfer limit is reached (this should allow for 75+ more full mirrors), or a month has passed, whichever comes first.

as-com commented 7 years ago

Second mirror: https://mirrors.asun.co/climate-mirror/aqsdr1.epa.gov/aqsweb/aqstmp/airdata/ hashdeep: https://mirrors.asun.co/climate-mirror/aqsdr1.epa.gov/aqsweb/aqstmp/airdata/hashdeep.txt

CorentinB commented 7 years ago

I'm currently downloading it using lftp mirror.

x775 commented 7 years ago

I have a complete copy as of this posting.

md5: 368914f7fbc4f5d93771ae28783a02ff sha256: 7db788976a63088f2aa6f57b9ccefa736e819fc2fa510e39bd33a6bded16275a

Individual checksums: https://gist.github.com/x775/ddfbd273faeec0a610a8e4c297fca4c9 Direct download links: https://gist.github.com/x775/182339a9649cda0ded5d47c8276c17af

Size: 12.71862GB GB

Compressed name: EPA_Air_Quality_Data_Archives.7z Compressed md5: bf7445c9ada22a92aaca4dc3f737be2e
Compressed sha256: 94ab392f322c4c156349d4af86d50d5272d420716f32990b40a5ea8d50ca01b8 Compressed size: 9.04773GB Compressed download link: https://drive.google.com/open?id=0B6PlQrUTwL1PU1FEQWdFMms1d0k