climate-mirror / datasets

For tracking data mirroring progress
201 stars 18 forks source link

EPA Air Quality System (AQS) #96

Open nickrsan opened 7 years ago

nickrsan commented 7 years ago

Name: Organization: EPA Description URL: https://www.epa.gov/aqs Download URL: https://aqs.epa.gov/aqsweb/documents/data_mart_welcome.html File Types: Size: Status:

ghost commented 7 years ago

I'll get this one. Looks like contacting the agency is only required when using their API. I'm going to get the pre-retrieved files here: https://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/download_files.html. It'll be on a private Amazon Cloud backup for now; I'll post the HTML version publicly-available soon.

lpiep commented 7 years ago

@blevins321, The pre-retrieved files only cover a subset of the database. The full Data Mart is 3.4 billion rows according to https://aqs.epa.gov/aqsweb/documents/data_mart_welcome.html.

lrehmann commented 7 years ago

Grabbed the pre-generated files and re-hosted at http://rehmann.co/blog/pre-generated-data-files-airdata-epa-mirror/ [13G] Still uploading locally to the server.

gpk320 commented 7 years ago

Here's some code I'm using to download columns from the EPA's http://iaspub.epa.gov/enviro/efservice database. Perhaps it can help. NOTE: Don't accidentally DOS yourself! The timeouts and sleeps are an important part of the code. https://bitbucket.org/azimuth-backup/azimuth-inventory/src/1c1fcc415dc240189bdcfb10ef1d6e44ab0118ad/www.epa.gov_climatechange/db_downloadTRI1.py?at=master&fileviewer=file-view-default