Update to Rwanda REMA data source

majesticio commented 6 months ago

Updates to our Rwanda source, uses a new data URL

caparker commented 6 months ago

A few things to do before we can push this one

I updated the way that we are specifying the units. Moving forward lets do it this way for all new/updated adapters.
Is there a way to get the siteid in there as the source_id? For example
```
"title": "Rusizi", "site_id": "RSZ"
```
Is there a reason that you are filtering out the zeros from the measurements?
Update the documentation for this adapter based on our new template. I would like to see more details about the units and why we have chosen the units we have. If there is data already in the database then lets look at the range of that data and include it.

I added a method that will summarize the parameters while doing the dry run, the output will look something like this

  {
    "message": "[Dry Run] New measurements found for rwanda-rema: 890",
    "failures": {},
    "count": 890,
    "duration": 0.117,
    "from": "2024-03-14T23:39:15.000Z",
    "to": "2024-03-14T23:39:15.000Z",
    "parameters": {
      "co": {
        "min": 0,
        "max": 766.23,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "no2": {
        "min": 0,
        "max": 38.01,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "o3": {
        "min": 0,
        "max": 57.47,
        "nulls": 0,
        "errors": 0,
        "count": 140
      },
      "pm10": {
        "min": 0,
        "max": 71.46,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "pm25": {
        "min": 0,
        "max": 60.71,
        "nulls": 0,
        "errors": 0,
        "count": 150
      },
      "so2": {
        "min": -218.94,
        "max": 59.97,
        "nulls": 0,
        "errors": 0,
        "count": 150
      }
    },
    "sourceName": "rwanda-rema"
  }
]

That should help with figuring out the units.

majesticio commented 6 months ago

That all sounds good. I was seeing a lot of zero values being returned which I interpreted to mean that there was no data available, which was assumption on my part. I will stop filtering for 0 values.

majesticio commented 6 months ago

Requested changes have been made and merge conflicts resolved

majesticio commented 6 months ago

Added a try/catch block as requested

majesticio commented 5 months ago

There is an issue with time stamps being ahead with this adapter. The provided timestamps eg 2024-04-16T17:32:45Z imply that this is UTC time. However, the resulting measurements are slightly ahead of current time. Additionally, looking at the timestamps ignoring UTC timezone, they align completely with the local time in Rwanda.

I have updated the code to compensate for this discrepancy and will add a note in the documentation

openaq / openaq-fetch

Update to Rwanda REMA data source #1089