A Krawler based service to download data from OpenAQ
The k-openaq allows to download OpenAQ data for a given list of countries. The available countries is available here. Also, a full Json output can be retrieved using the the following query
The available variables are:
pm25
: particulate matter PM255pm10
: Particulate matter PM10so2
: Sulphur dioxide no2
: Nitrogen dioxide o3
: Ozoneco
: Carbon monoxide bc
: Black carbonThe job relies on the OpenAQ API v2.0.
[!NOTE] Because of Rate limiting It is recomended to register to OpenAQ to get an API Key.
The following diagram illustrates how the job works:
The job can be configured by setting the following environment variables:
Parameter | Description |
---|---|
API_KEY |
the API Key. |
COUNTRIES_IDS |
an array to specify the countries to take into account. By default: '2', '134','133','132' |
VARIABLES |
an array to specify the variables to scrape. By default: 'pm25', 'pm10', 'so2', 'no2', 'o3', 'co', 'bc' |
QUERY_LIMIT |
the limit of returned results. By default: 1000 |
TTL |
the time to live of the data. By default: 7 day |
TIMEOUT |
the timeout of the request. By default: 60 * 60 * 1000 |
DB_URL |
the database url. By default: mongodb://localhost:27017/openaq |
We personally use Kargo to deploy the service.
Please refer to contribution section for more details.
This project is sponsored by
This project is licensed under the MIT License - see the license file for details