Shape fill shape background instead of shape color, and black with missing data

[bijwaard]

I added fillColor to the panel config, on the "DSO connection" on top left and inverters on the bottom. I expected the shape to be filled with the fillColor, but instead the background of the shape image gets filled with the color.

  DSO Generation:
    dataRef: "CT:sum-active"

    fillColor:
      gradientMode: "hue"
      thresholds:
        - {color: "semi-dark-green", level: 0}
        - {color: "orange", level: 500}
        - {color: "red", level: 1000}

The style of the shape is as follows: pointerEvents=1;verticalLabelPosition=bottom;shadow=0;dashed=0;align=center;fillColor=rgba(50, 172, 45, 0.97);html=1;verticalAlign=top;strokeWidth=1;shape=mxgraph.electrical.signal_sources.ac_source;strokeColor=#000000;fontColor=#000000;fontSize=14;fontStyle=1

Furthermore, when data is missing, the shape background becomes black and all other labels are missing in the figure. I would rather use the last known data value, since not all data may always be nicely linked up.

Kind regards, Dennis

[bijwaard]

The shape fill does appear to work with the voltage shape, for this I now have site config:

---

#------------------------------------------------------------------------------
# YAML Aliases to simplify maintenance
thresholds:
  powerThresholds:
    - {color: "semi-dark-green", level: 0}
    - {color: "orange", level: 500}
    - {color: "red", level: 1000}
  voltageThresholds:
    - {color: "#semi-dark-green", level: 0}
    - {color: "orange", level: 245}
    - {color: "red", level: 250}

And panel config:

  VT:L1:
    dataRef: "VT:1"
    label:
      separator: "replace"
      units: "volt"
      decimalPoints: 1
    labelColor:
      gradientMode: "none"
      thresholdsRef: "voltageThresholds"
    fillColor:
      gradientMode: "hue"
      thresholdsRef: "voltageThresholds"
  VT:L2:
    dataRef: "VT:2"
    label:
      separator: "replace"
      units: "volt"
      decimalPoints: 1
    labelColor:
      gradientMode: "none"
      thresholdsRef: "voltageThresholds"
    fillColor:
      gradientMode: "hue"
      thresholdsRef: "voltageThresholds"
  VT:L3:
    dataRef: "VT:3"
    label:
      separator: "replace"
      units: "volt"
      decimalPoints: 1
    labelColor:
      gradientMode: "none"
      thresholdsRef: "voltageThresholds"
    fillColor:
      gradientMode: "none"
      thresholdsRef: "voltageThresholds"

The shape style of the voltage shape is: perimeter=ellipsePerimeter;verticalLabelPosition=bottom;shadow=0;dashed=0;align=center;fillColor=rgba(50, 172, 45, 0.97);html=1;verticalAlign=top;strokeWidth=1;shape=mxgraph.electrical.instruments.voltmeter;strokeColor=#000000;fontColor=#000000;fontSize=14;fontStyle=1

The voltage panel looks like this, however I expected to see some semi-dark-green for voltage L1, L2, and L3, while L1 and L2 get orange and L3 gets black with gradientMode "None" (The L3 value is available though):

[andymchugh]

ok, there's a few things to unpack in here:

• yaml has reserved characters like ':', but it is a super set of json. So if you need the ':' character in your keys you should be able to quote the key, just as you would have to in json. I'm not saying this is related, just an fyi as the yaml keys look odd with a ':' in them and unquoted so that might be contributing to the problem.
• re where the fill color gets applied, the panel looks up the element-id you've defined in yaml and then in the svg dom recurses in storing away the elements so that it can in the render pass apply the correct label/fill color on the whole element tree. Quite possibly it's not doing the right thing with all widgets. The only way I can look at it though is to have an example svg that demonstrates the problem. If you also add a yaml leveraging the test-data series the panel creates it will make it as simple as possible to see the problem.
• re the null data and whether the panel should support 'last-not-null', that's more nuanced. Playing devils advocate on it I think there's two query functions that are relevant. keepLastValue(INF) can be added to the query to effectively mimic 'last-not-null'. But even better, that also allows you to time-limit it. i.e. not lock in the last value that could be quite historical and cause you to misread what's happening now. The other one is transformNull(0). That one I use a lot with rates because zero flow results in a null rate. I think use of these functions in the queries probably makes more sense than supporting 'last-not-null', especially given the timeSlider. But I just want to put these ideas out there so you can respond. If they don't work for you it would be good for me to understand how your usecase differs.
• last one is what happens to the fill if there is a null value. That is looking like a bug. In that I'd expect it not to drive the color but instead it appears to be driving it black. I'll start looking into this one.

So a few points, do come back and say if I'm misunderstanding where you're coming from or have missed any points you are making?

[bijwaard]

Thanks for your quick response!

* yaml has reserved characters like ':', but it is a super set of json. So if you need the ':' character in your keys you should be able to quote the key, just as you would have to in json. I'm not saying this is related, just an fyi as the yaml keys look odd with a ':' in them and unquoted so that might be contributing to the problem.

I first had success with quoting the ":" with a backslash "\:", later it started working without for some reason, even spaces in the keys work as long as they are not preceded with a ":". The YAML key / value separator seems to be ": ".

* re where the fill color gets applied, the panel looks up the element-id you've defined in yaml and then in the svg dom recurses in storing away the elements so that it can in the render pass apply the correct label/fill color on the whole element tree. Quite possibly it's not doing the right thing with all widgets. The only way I can look at it though is to have an example svg that demonstrates the problem. If you also add a yaml leveraging the test-data series the panel creates it will make it as simple as possible to see the problem.

I can reproduce the shape background problem by just changing the style (see message above) on the right-most element of your default SVG and data. Here is the updated drawio/SVG file:

* re the null data and whether the panel should support 'last-not-null', that's more nuanced. Playing devils advocate on it I think there's two query functions that are relevant. **keepLastValue(INF)** can be added to the query to effectively mimic 'last-not-null'. But even better, that also allows you to time-limit it. i.e. not lock in the last value that could be quite historical and cause you to misread what's happening now. The other one is **transformNull(0)**. That one I use a lot with rates because zero flow results in a null rate. I think use of these functions in the queries probably makes more sense than supporting 'last-not-null', especially given the timeSlider. But I just want to put these ideas out there so you can respond. If they don't work for you it would be good for me to understand how your usecase differs.

The keepLastValue sounds like a nice way, but cannot find how to use this in an influxql query, or is this something you can enable within your plugin, or is it Graphite specific? I was also playing with the "query option" to increase the minimal time interval to more than a few seconds, this partly solves this, but does not always work for the first and last slider position (or "now" without slider). In my case the data comes from voltage and current sensors, their values vary quite a lot during the day, changing Null to a fixed value like 0 would disturb the series.

* last one is what happens to the fill if there is a null value. That is looking like a bug. In that I'd expect it not to drive the color but instead it appears to be driving it black. I'll start looking into this one.

Hope you can easily find what is driving Null values to black shape color.

So a few points, do come back and say if I'm misunderstanding where you're coming from or have missed any points you are making?

My use-case may indeed be somewhat different from yours, your plugin seems to quite general purpose:-)

Kind regards, Dennis

[bijwaard]

Hi Andy,

I found the part of the style that may trigger the rectangle background around the ellipse shape, this seems to be triggered by the "pointerEvents=1;". I removed this part of the shape style and now the background color is confined to the ellipse, wouldn't have guessed based on the style property name.... Unsure how this was solved in flowcharting plugin, there background colors worked correctly.

Just compared working style with the non-working style for educated guess;-) Didn't find pointerEvent description in drawio shape style faq

Kind regards, Dennis

[bijwaard]

There also seems to be a problem with the foreground color of the voltage shape, the black V in the circle that is in the original turns to background color when the plugin fills the background. I remember that flowcharting uses a draw.io API to manipulate the shape background, are you also using a specific API to manipulate the shapes?

Here is the part of the shape that draws the letter V in the foreground (Style panel / Edit / Edit Style):

  <background>
    <ellipse h="90" w="90" x="0" y="0" />
  </background>
  <foreground>
    <fillstroke />
    <fontsize size="50" />
    <text align="center" str="V" valign="bottom" x="45" y="70" />
  </foreground>

The plugin turns this into (without the V):

In SVG, this shape is drawn as follows, with "V" inside a HTML <text/> within a group that sets the font and fill color, could this fill color be overridden with the background fill color?

<g id="cell-VT:L1">
<ellipse cx="25" cy="198.5" rx="22.5" ry="22.5" fill="rgba(50, 172, 45, 0.97)" stroke="#000000" pointer-events="all"/>
<g fill="#000000" font-family="Arial,Helvetica" text-anchor="middle" font-size="25px">
<text x="24.5" y="208.5">
V</text>
</g>
<g transform="translate(-0.5 -0.5)">
<switch>
<foreignObject pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility" style="overflow: visible; text-align: left;">
<div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 228px; margin-left: 25px;">
<div data-drawio-colors="color: #000000; " style="box-sizing: border-box; font-size: 0px; text-align: center;">
<div style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; pointer-events: all; font-weight: bold; white-space: nowrap;">
242.8 V</div>
</div>
</div>
</foreignObject>
<text x="25" y="242" fill="#000000" font-family="Helvetica" font-size="14px" text-anchor="middle" font-weight="bold">
242.8 V</text>
</switch>
</g>
</g>

The other shape with the "~" in the circle is a drawing inside the shape, which stays visible.

  <background>
    <ellipse h="60" w="60" x="0" y="0" />
  </background>
  <foreground>
    <fillstroke />
    <path>
      <move x="15" y="30" />
      <arc large-arc-flag="0" rx="8" ry="8" sweep-flag="1" x="30" x-axis-rotation="0" y="30" />
      <arc large-arc-flag="0" rx="8" ry="8" sweep-flag="0" x="45" x-axis-rotation="0" y="30" />
    </path>
    <stroke />
  </foreground>

[andymchugh]

I'm going to treat this issue as the following:

• lastNotNull support wanted because some data sources don't have the graphite keepLastValue query function available.
• When the data is null, the fill color is still being driven when it shouldn't be. I'll look into that and sort it out if I find a problem.

Re the shape fill color problems, it's all a bit too complex to take forward in this form. Rectangle fill colors work correctly. database fill colors work correctly. I'm sure circle and ellipse fill colors also work correctly. And that range of shapes seems to give you the support you need. If you still think this is a problem for you, can you raise a separate issue with an attached full svg that demonstrates the problem. I'm open to making it as robust as possible but whilst the problem doesn't appear blocking it's going to be a lower priority.

[bijwaard]

Great solution for now, thanks!

Because of the number of drawio shapes, there may be more shapes that have background/foreground issues like described above. I'll raise a new issue with SVG file example when it gets out of control;-)

[bijwaard]

Small update: I updated the voltage shape to use line-paths instead of the letter V to keep using the shape within the flow panel.

[andymchugh]

As of 1.6.0 lastNotNull can be configured in the yaml to make sparse data look solid. And when data is null, the color drives now drive to the original fill and label colors. So should be good now. https://grafana.com/grafana/plugins/andrewbmchugh-flow-panel/

[bijwaard]

I found the following config in the updated panelConfig YAML, testDataSparse does not seem to enable lastNotNull for metrics from other datasources? Or is there another configuration for that? With v1.6.0, the colors now seem to revert to the original color from the SVG when data is NULL, no longer black.

# Version 1.6.0 onwards: This section details test settings used in validating the functionality.
# They provide a way to drive particular code pathways.
test:
  # This changes the test timeSeries to be sparse and is used as a way to demonstrate the
  # difference between datapoint 'last' and 'lastNotNull'. Undefined defaults to false.
  testDataSparse: false

[andymchugh]

I found the following config in the updated panelConfig YAML, testDataSparse does not seem to enable lastNotNull for metrics from other datasources? Or is there another configuration for that? With v1.6.0, the colors now seem to revert to the original color from the SVG when data is NULL, no longer black.

# Version 1.6.0 onwards: This section details test settings used in validating the functionality.
# They provide a way to drive particular code pathways.
test:
  # This changes the test timeSeries to be sparse and is used as a way to demonstrate the
  # difference between datapoint 'last' and 'lastNotNull'. Undefined defaults to false.
  testDataSparse: false

Sorry, should have included the config reference. The config to drive lastNotNull can be set at the panel level, the cell level or the cell attribute level. See:

https://github.com/andymchugh/andrewbmchugh-flow-panel/blob/main/yaml_defs/panelConfig.yaml#L90

https://github.com/andymchugh/andrewbmchugh-flow-panel/blob/main/yaml_defs/panelConfig.yaml#L125

https://github.com/andymchugh/andrewbmchugh-flow-panel/blob/main/yaml_defs/panelConfig.yaml#L140 ...

[bijwaard]

great! That appears to work nicely on the panel level didn't manage to trigger it at deeper level yet...

# This is the panel level default which when undefined defaults to 'last'. It can be overriden at the cell
# level and the cell-attribute level.
#
# Valid values are undefined, "last", "lastNotNull"
datapoint: "lastNotNull"

cellIdPreamble: "cell-"
cells: 
  ...