k-openaq
A Krawler based service to download data from OpenAQ
Description
The k-openaq allows to download OpenAQ data for a given list of countries. The available countries is available here. Also, a full Json output can be retrieved using the the following query
The available variables are:
pm25: particulate matter PM255pm10: Particulate matter PM10so2: Sulphur dioxideno2: Nitrogen dioxideo3: Ozoneco: Carbon monoxidebc: Black carbon
The job relies on the OpenAQ API v2.0.
[!NOTE] Because of Rate limiting It is recomended to register to OpenAQ to get an API Key.
The following diagram illustrates how the job works:
Configuration
The job can be configured by setting the following environment variables:
| Parameter | Description |
|---|---|
API_KEY |
the API Key. |
COUNTRIES_IDS |
an array to specify the countries to take into account. By default: '2', '134','133','132' |
VARIABLES |
an array to specify the variables to scrape. By default: 'pm25', 'pm10', 'so2', 'no2', 'o3', 'co', 'bc' |
QUERY_LIMIT |
the limit of returned results. By default: 1000 |
TTL |
the time to live of the data. By default: 7 day |
TIMEOUT |
the timeout of the request. By default: 60 * 60 * 1000 |
DB_URL |
the database url. By default: mongodb://localhost:27017/openaq |
Deployment
We personally use Kargo to deploy the service.
Contributing
Please refer to contribution section for more details.
Authors
This project is sponsored by
License
This project is licensed under the MIT License - see the license file for details