No data ("out of compliance") buildings and polygon style

[tomay]

Current situation

GHGs and GHG Intensity:

• Blue indicates "no data" but these often actually have data:

• "True" no data:

Energy Performance Metrics

No data is symbolized with a grey footprint:

Goals

1. One nodata symbol for all map types
2. Understand why some "blue" buildings, above, actually do have data
3. Add text to building report about no data

[tomay]

Some relevant code:

• No data test: https://github.com/GreenInfo-Network/seattle-building-dashboard/blob/master/src/app/views/map/building_layer.js#L120-L122
• CartoCSS generator: https://github.com/GreenInfo-Network/seattle-building-dashboard/blob/master/src/app/views/map/building_layer.js#L541-L585
• Model for calculating buckets, etc. https://github.com/GreenInfo-Network/seattle-building-dashboard/blob/master/src/app/models/building_color_bucket_calculator.js

Color range defs:

[tomay]

Trying to understand how this one building is getting a blue footprint

It shows 276.8 for total_ghg_emissions

And the generated CartoCSS looks like this:

#seattle_buildings_2021_update {polygon-fill: #CCC;polygon-opacity: 0.9;line-width: 1;line-color: #FFF;line-opacity: 0.5;}
#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#1f5dbe}
#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#306fa5}
#seattle_buildings_2021_update [total_ghg_emissions>=2.2]{polygon-fill:#40808c}
#seattle_buildings_2021_update [total_ghg_emissions>=2.8333333333333255]{polygon-fill:#519273}
#seattle_buildings_2021_update [total_ghg_emissions>=3.7]{polygon-fill:#6ba165}
#seattle_buildings_2021_update [total_ghg_emissions>=5.022222222222228]{polygon-fill:#90ae60}
#seattle_buildings_2021_update [total_ghg_emissions>=7.3]{polygon-fill:#b5bb5b}
#seattle_buildings_2021_update [total_ghg_emissions>=12.5]{polygon-fill:#dac857}
#seattle_buildings_2021_update [total_ghg_emissions>=18.3]{polygon-fill:#ffd552}
#seattle_buildings_2021_update [total_ghg_emissions>=25.4]{polygon-fill:#f7c34e}
#seattle_buildings_2021_update [total_ghg_emissions>=34.344444444444456]{polygon-fill:#efb24a}
#seattle_buildings_2021_update [total_ghg_emissions>=46.7]{polygon-fill:#e6a045}
#seattle_buildings_2021_update [total_ghg_emissions>=60.4]{polygon-fill:#de8f41}
#seattle_buildings_2021_update [total_ghg_emissions>=77.07777777777778]{polygon-fill:#d57b3c}
#seattle_buildings_2021_update [total_ghg_emissions>=103.2]{polygon-fill:#ca6537}
#seattle_buildings_2021_update [total_ghg_emissions>=139.0666666666669]{polygon-fill:#c04f32}
#seattle_buildings_2021_update [total_ghg_emissions>=209.65555555555542]{polygon-fill:#b5392d}
#seattle_buildings_2021_update [total_ghg_emissions>=356.8111111111113]{polygon-fill:#ab2328}

Which means, the color should be: #b5392d

Could it have something to do with the first two lines that both evaluate values >=0?

#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#1f5dbe}
#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#306fa5}

I don't think so, otherwise all buildings would be affected

[tomay]

I set up a locally running example, based on this, with the following addition in done:

          cartodb.createLayer(map, {
            user_name: 'cityenergy-seattle',
            type: 'cartodb',
            sublayers: [
              { "sql": "SELECT a.*,b.total_ghg_emissions,b.total_ghg_emissions_intensity,b.site_eui_wn,b.energy_star_score,b.neighborhood,b.councildistrict,b.reported_gross_floor_area,b.yearbuilt,b.id FROM seattle_building_outlines_2021 a,seattle_buildings_2021_update b WHERE a.buildingid = b.id AND b.year=2021",
                "cartocss": "#seattle_buildings_2021_update {polygon-fill: #CCC;polygon-opacity: 0.9;line-width: 1;line-color: #FFF;line-opacity: 0.5;}\n#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#1f5dbe}\n#seattle_buildings_2021_update [total_ghg_emissions>=0]{polygon-fill:#306fa5}\n#seattle_buildings_2021_update [total_ghg_emissions>=2.2]{polygon-fill:#40808c}\n#seattle_buildings_2021_update [total_ghg_emissions>=2.8333333333333255]{polygon-fill:#519273}\n#seattle_buildings_2021_update [total_ghg_emissions>=3.7]{polygon-fill:#6ba165}\n#seattle_buildings_2021_update [total_ghg_emissions>=5.022222222222228]{polygon-fill:#90ae60}\n#seattle_buildings_2021_update [total_ghg_emissions>=7.3]{polygon-fill:#b5bb5b}\n#seattle_buildings_2021_update [total_ghg_emissions>=12.5]{polygon-fill:#dac857}\n#seattle_buildings_2021_update [total_ghg_emissions>=18.3]{polygon-fill:#ffd552}\n#seattle_buildings_2021_update [total_ghg_emissions>=25.4]{polygon-fill:#f7c34e}\n#seattle_buildings_2021_update [total_ghg_emissions>=34.344444444444456]{polygon-fill:#efb24a}\n#seattle_buildings_2021_update [total_ghg_emissions>=46.7]{polygon-fill:#e6a045}\n#seattle_buildings_2021_update [total_ghg_emissions>=60.4]{polygon-fill:#de8f41}\n#seattle_buildings_2021_update [total_ghg_emissions>=77.07777777777778]{polygon-fill:#d57b3c}\n#seattle_buildings_2021_update [total_ghg_emissions>=103.2]{polygon-fill:#ca6537}\n#seattle_buildings_2021_update [total_ghg_emissions>=139.0666666666669]{polygon-fill:#c04f32}\n#seattle_buildings_2021_update [total_ghg_emissions>=209.65555555555542]{polygon-fill:#b5392d}\n#seattle_buildings_2021_update [total_ghg_emissions>=356.8111111111113]{polygon-fill:#ab2328}",
                "interactivity": "id,id"
              }
            ]
          }, { https: true }).addTo(map);

That works (when you also set the map to focus on Seattle), but so far, it is not giving me any insight into why this is not working. Something wrong with the CartoCSS? Something wrong with the values in the table?

[tomay]

Okay, so I'm getting somewhere. This one has two buildings that share the same footprint and have different names:

SKINNER BLDG

COBB APARTMENTS

For some reason, I'm not able to get that "no data" to show on top when switching between these two, but clearly it is possible as seen in the screenshot up channel

So that explains at least that one error....

[tomay]

The second source of confusion seems to be that you can visualize the map based on total_ghg_emissions or total_ghg_emissions_intensity, and these can be 0, but you can also have a non-zero site_eui_wn which is what is shown prominently on the popup. Maybe that shouldn't be true but it is

2021 has 40 of these, where total_ghg_emissions = 0 and site_eui_wn is > 0:

And (for me at least) the fact that total_ghg_emissions and total_ghg_emissions_intensity are not shown on the popup when they are on the map, is confusing

So I believe that solves Goal 2 from the top of this issue

[tomay]

Remaining to do per Mike's request last check-in:

• [x] Communicate to Mike that buildings can (and man do) have no data for total_ghg_emissions and total_ghg_emissions_intensity but still show data for site_eui_wn in the popup
• [ ] Consider changing all blue nodata footprints to grey (Mike wanted "grey hatch" which might be possible with CartoCSS but that sets up another CARTO dependency). I think this is easy but I haven't looked into it. We can pattern this after the grey nodata already in place for the other metrics.
• [ ] Mike also wants to add conditional "no data" language on the building reports

[tomay]

Working on the second item in the checklist above ("Consider changing all blue nodata footprints to grey"), not as straightforward as you might think:

• CartoCSS marker color defaults to #CCCCCC and then generates additional colors based on the color_range and filter_range specified in seattle.json.
• The ranges for total_ghg_emissions and total_ghg_emissions_intensity start at 0, and, as seen above, the buildings with what I assume to be "No Data" report a value of 0, not null or blank.
• The range for energy_star_score starts at 1, and these get actual null values as seen here:

So I think the resolution to this issue looks something like this:

• Data should be encoded with actual null values or blanks for buildings missing total_ghg_emissions and total_ghg_emissions_intensity. If that's hard or not desired, then we could just make the first color #CCCCCC noting that this will be stretched to the next color across values, as seen here:

The blue never shows in the chart on the left, which admittedly is a little hard to understand

• We apply a new color_range for those two measures that does not include the blue #1F5DBE as the initial value (matching 0, true 0, not "No Data"). We could just copy in the same values that are used for energy_star_score if that fixes it. Or come up with some new ramp. NOTE: site_eui_quartiles
• Note: "Building EUI Quartiles" also uses this blue #1F5DBE for the first bin, do we keep that?
• Final note, the use of this blue seems quite intentional, so I think we should mention that and discuss what the intent may have been before scrapping it.

[tomay]

Mike has said several times he wants to use a hatch for the out of compliance buildings (e.g. where GHG emissions, GHG intensity, Site EUI, Building EUI is null).

This will have to be some kind of non-standard CartoCSS style.

According to this comment, from a CARTO employee, in 2018 the only way to do this is to upload your own pattern:

CARTO does not provide a collection of existing pattern, you have two options: Use external urls: they have to be publicly available and stable Upload you own patterns [to CARTO]: you need to use the IMG button on a point layer, that interface allows you to upload your own images, so then you can get the final URL and use it in your own polygon layer.

Also, currently building_layer.js defines a baseCartoCSS that is the same for every filter type. This request breaks that model, now we not only have a different marker-fill (in the case of points) or polygon-fill (for footprints) for different filters, but we also need to provide a hatch style, that only applies to null values

One idea, though it's very messy, could be quick:

• Define the hatch pattern and pattern opacity for footprints in baseCartoCSS. If the selected filter is not of a type that defines a hatch, then overwrite pattern opacity (set it to 0).
• Also, always set pattern opacity to 0 for every additional class written to the cartoCSS file

[tomay]

Okay, I've got a hatch style working

This new approach stomps all over the assumptions built into the original implementation, which has a base style in one file (building_layer.js), isolated from a calculator to construct the dynamic CartoCSS styles in another file (building_bucket_calculator.js)

This is because the hatch styles bleed into both files - we need a conditional basestyle with opacity declared in building_layer.js - and we also need CSS declarations in every other line of CartoCSS, in order to explicitly make this polygon image transparent for all other non-null classes (and only for polygons).

This approach conditionally applies the hatch to different map layers, with the use of a new map_layers declaration of hatch_null_css in seattle.json:

    "map_layers": [
        {
            "title": "Total Seattle GHG Emissions",
            "field_name": "total_ghg_emissions",
            "display_type": "range",
            "range_slice_count": 18,
            "section": "Greenhouse Gas Emissions",
            "color_range": ["#1f5dbe","#599b67","#ffd552","#da863f","#ab2328"],
            "hatch_null_css": true,
            "unit": "Metric Tons CO₂e",
            "filter_range": {"min" : 0, "max" : 500},
            "formatter": "fixed-1",
            "description": "Greenhouse Gas (GHG) Emissions ...

For now, I'm hosting this hatch image on an S3 bucket on Greeninfo's AWS account: https://s3.console.aws.amazon.com/s3/upload/seattle-buildings-polygon-hatch-images?region=us-west-1

[tomay]

Note: hatch only works on Site EUI (WN) and Building EUI Quartiles. It won't show up on the GHG Filters until those values are updated to include actual null values

Side note: Another surprising find here is that Building EUI Quartiles in seattle.json uses site_eui_wn, instead of building_type_eui_wn....

        {
            "title": "Building EUI Quartiles",
            "id": "site_eui_quartiles",
            "field_name": "site_eui_wn",

[tomay]

Last item: Why do the histograms not show the full color range of the dots and footprints?

There's this cryptic note in building_color_bucket_calculator.js (one of the few mutli-line comments in this entire code base):

  BuildingColorBucketCalculator.prototype.colorGradient = function() {
    if (this.memoized.colorGradients.hasOwnProperty(this.fieldName)) {
      return this.memoized.colorGradients[this.fieldName];
    }

    // This is how we calculate the colors for the dots on the map.
    // But they don't line up with the colors in the histogram. Why not?

    // The domain is "fieldValues", which is an unordered list of all of the building value for this field.
    // But the domain for the histogram color ramp is just linear max and min for the given field.
    // And more importantly, it needs to be the max and min that's set according to the config file. That's how the colors get determined in the histogram,

[tomay]

I moved the last comment to a new issue.

Otherwise, calling this done pending feedback from the client