Improving data acquisition

[calvinjlin]

Problem

At present, data files are downloaded manually. However, this can be labor intensive/time consuming for the programmer. Looking at the cookiecutter project structure as well as other IEL, data retrieval via scripts/apis would probably be a better approach.

PurpleAir Solution: Access Data Through Thingspeak

Currently, PurpleAir data files are retrieved using the online download tool. However, this can necessitate a lot of button clicking (16 sensors, each sensor has 4 files due to A/B channels and primary/secondary datasets). PurpleAir provides other ways to access data. A better approach would be to utilized the Thingspeak API.

1. Retrieve Thingspeak API keys and channel IDs for each sensor (also labor intensive).

   • [x] Use urllib2 + beautifulsoup to extract 5-digit PurpleAir identifier from "Show on map" hyperlinks on sensor download page.
   • [x] Extract Thingspeak API keys and channel IDs from JSON file located at https://www.purpleair.com/json?show=[PurpleAir_ID]
   • [x] Save Thingspeak API keys and channel IDs to excel file

2. Implement Thingspeak API for data retrieval.

   • [x] Review API docs . . .

TCEQ

(to be expanded)

• EPA AirNow has API that allows access to a few of TCEQ's sensors (only those approved for regulatory use).
• Prefer yearly data to monthly data

[calvinjlin]

By using Firefox's developer tools, I found a GET request url leading to a json with the metadata for all 16 sensors. Therefore, the json module was used instead of beautiful soup. It has the Thingspeak IDs and also other metadata like latitude and longitude.