openaq / openaq-fetch

A tool to collect data for OpenAQ platform.
MIT License
84 stars 39 forks source link

recover new south wales #1107

Open majesticio opened 5 months ago

majesticio commented 5 months ago

This is a draft. it appears to be returning valid measurements but is not finishing:

info: 0 new measurements found for "Australia - New South Wales" in 1.553s
info: Australia - New South Wales - 1 occurrences of Adapter error (au.json/nsw/'Australia - New South Wales'): Cannot read properties of undefined (reading 'readable')
caparker commented 5 months ago

I got it working and did some cleanup but there is still work to do on the parameters + units.

A few notes that I wanted to share You were catching errors in both the fetchStations and the fetchMeasurements methods and then not rethrowing them. Each of those methods must run successfully in order for the how adapter to work. Given that and based how you designed this I would have expected either

  1. No error catching in either of those methods. The try/catch in the fetchData method would catch it and we would just have to hope that the message returned was indicative of what method it happened in.
  2. Catch the error in each method but then rethrow it with more detail about where it happened.

Also the repetitive use of find can be fixed by objectifying the array and keying it on Site_Id.

Make sure that all the units are units we allow and if they are not we will need to fix them.

russbiggs commented 5 months ago

This needs quite a bit more work in the documentation doc before proceeding with development. Once documentation is more thorough we can proceed

majesticio commented 5 months ago

I corrected the use of the timezone ('Australia/Sydney'). The source data date times are all in local time. I made note of it in our documents for future reference