possible bug: gradual decrease in number of stations

[hydrosquall]

Hi,

When I started following this project in #2 , the tables had data from 5 stations. A few weeks later, it went up to 6 with the addition of cross-bronx expressway.

https://github.com/nychealth/realtime-air-quality/blob/6eec57d745898fdb4962459edea90d48d61b6397/RT_flat.csv#L1-L2

A little over a month ago, it dropped to 4 stations

https://github.com/nychealth/realtime-air-quality/blob/089b00105e308a7d9b847049692734ae642be126/RT_flat.csv#L1-L2

However, we're currently only reporting data from 2 stations

https://github.com/nychealth/realtime-air-quality/blob/b1a869aaccf02a989571bcf142ee37b8e4ca9d83/RT_flat.csv#L1-L2

If you use the graphs on the public hub, you'll only see 1 station reported since it's hard-coded to work with Queens College but not Midtown-DOT.

I wanted to check if the sensors that were removed from the CSVs are permanently gone, or if there might be a code change that could be made to bring the data from the old sensors back. I'm hoping that this is a temporary adjustment related to upgrading old sensors to use new hardware.

Thanks for making this important air pollution data public!

[cgettings]

Hi @hydrosquall,

Thanks for the question, and thanks again for your interest.

I talked to @chuskey79 about the monitors a couple weeks ago, and he mentioned that we were waiting on some contractors to do something to get them up again (he was more specific than that, but I can't remember the details).

Regarding the code, I’ve been neglecting this branch in my development, and instead working in this repo, where we’re developing the new version of the EHDP, including the AQ hub (though this week we’re migrating everything here). The current live version on the hub is an adaptation of this prototype. You can find the actively developed code here and here.

Regarding the data updates, we’ve actually switched over to using an Azure endpoint for the portal prototype, as well as the current live site, as detailed in issue #229 in that repo. This repo is still being updated because I only made that change to the current site last week. We can probably archive this (attn: @mmontesanonyc).

Regarding the station hard-coding, our list of to-dos includes refactoring the code to use long-format data, so that it can be agnostic w/r/t the number and names of stations. More elegant and less problematic. In the meantime, I’ll check in with @chuskey79 to see what the station situation is, and edit the code to include all the live stations.

Thanks again!

[hydrosquall]

@cgettings , thanks for the explanation, excited to see the new EHDP site up and running! I used this data stream in an NYC open data week presentation in the spring. If I redo this presentation in a future season, I'll aim to point to the new portal instead.

our list of to-dos includes refactoring the code to use long-format data, so that it can be agnostic w/r/t the number and names of stations

In the short term, I rewrote the vega-lite spec for the chart to convert the wide to long data internally. (A section of my slides discusses the advantages of being able to do lightweight data transforms inside your charting grammar without needing to change upstream scripts). You still end up needing to supply the list of station names, but it's less verbose than creating a new vega layer for every station.

I see that several of the new site graphs are made with Datawrapper instead of Vega-Lite, but I'll share it just in case it may save you some time.

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "Real-time air quality",
  "width": "container",
  "height": "container",
  "padding": {"left": 0, "top": 10, "right": 0, "bottom": 25},
  "data": {
    "url": "https://raw.githubusercontent.com/nychealth/realtime-air-quality/6eec57d745898fdb4962459edea90d48d61b6397/RT_flat.csv"
  },
  "transform": [
    {
      "fold": [
        "Queensboro Bridge",
        "Williamsburg Bridge",
        "Cross Bronx Expy",
        "Queens College",
        "Manhattan Bridge",
        "Broadway/35th St"
      ],
      "as": ["location", "quality_reading"]
    }
  ],
  "layer": [
    {
      "mark": {
        "type": "line",
        "point": false,
        "tooltip": true,
        "interpolate": "monotone",
        "strokeWidth": 1.5
      },
      "encoding": {
        "x": {"field": "starttime", "type": "temporal", "title": ""},
        "y": {"field": "quality_reading", "type": "quantitative", "title": " "},
        "color": {"field": "location"}
      }
    }
  ]
}

Live Demo