search

openaq / openaq-fetch

A tool to collect data for OpenAQ platform.
MIT License
85 stars 39 forks source link

malfunctioning adapters #917

Open majesticio opened 2 years ago

majesticio commented 2 years ago

Adapters that need attention

This is a list of adapter names that are not returning data properly with their known errors. Does not include ones set as 'active : false' @caparker

caparker commented 2 years ago

Great list. I am checking each of them against the lambda branch and then will make a new issue for each so that they can be fixed and/or closed individually.

caparker commented 2 years ago

Australia - South Australia Added issue #918

caparker commented 2 years ago

Australia - Victoria Working on the lambda branch

node index.js -e .env -ds 'Australia - Victoria'
{"level":"info","message":"--- Dry run for Testing, nothing is saved to the database. ---"}
{"level":"info","message":"Looking up source Australia - Victoria"}
{"level":"info","message":"[Dry Run] File would be saved to realtime/2022-10-27/1666879891.ndjson"}
{"level":"info","message":"New measurements found for \"Australia - Victoria\": 68 in 0.048s"}
{"level":"info","message":{"errors":{},"itemsInserted":68,"results":[{"count":68,"duration":0.048,"failures":{},"message":"[Dry Run] New measurements found for Australia - Victoria: 68","sourceName":"Australia - Victoria"}],"timeEnded":1666879943839,"timeStarted":1666879891091}}
{"level":"info","message":"Dry run ended."}

Though I will say it takes up to a minute to get the response and that is not reflected in that log message above, it is correct in hte timeEnded - timeStarted but that doesnt seem to be reported. I recommend that we create an warning code for long adapters given that we only have 15min to work with in the lambda. The warning would help us identify the sources that we need to pull out to their own lambda.

We could also update this adapters error handling We could also merge this adapter with the other australian adapters because on first glance they seem pretty similar, differing only in config. Could be wrong there though :)

caparker commented 2 years ago

Bosnia_Tuzlanski Issue #919

majesticio commented 1 year ago

@caparker I have been going through the backlog of issues, it seems that issue #690 may be why there is a lot of data missing from EEA adapters (Germany is mentioned). I will be looking into this but please let me know if you have any thoughts.