Add Sparkline Graph tool

AmoebeLabs commented 1 year ago

The Problem To Be Solved

Current @2023.07.30!!!

Tool type: sparkline

Chart types and variants:

line
- variants:
- spline (currently line with smoothing enabled)
- stepline (NYI)
area (maybe real-time mode possible with fixed lower/upper bound??)
- variants:
- spline (currently line with smoothing enabled)
- stepline (NYI)
bar
equalizer
barcode
- variants:
- default (barcode)
- audio
- audio_top (NYI)
- audio_bottom (NYI)
radial barcode
- variants:
- default (radial barcode)
- sunburst_centered
- sunburst_inward
- sunburst_outward
- vizs:
- circle (default) (uses the radius of the circle)
- flower (rounded outward / rounded inward or outward/radius for background)
- rice_grain (rounded on both sides for both fore/background)
graded --> also works in real-time mode!
- variants:
- rank_order (currently named 'traffic light' with bin ordering)
- with value? As a bullet chart uses the current/target ?? The grading is the background in this case, or an extra horizontal line depicting the actual value within the grading range.

History...

Currently, there is a sparkline bar chart tool known as 'bar'.

Explore the possibilities of adding a Sparkline Graph tool, supporting for instance:

bar graph
line graph
line graph with dots/points
area graph (line graph with fill under) with line
dot plot (see: https://www.storytellingwithdata.com/blog/2020/12/9/what-is-a-dot-plot)
connected dot plot? (see above)

Questions to be explored if possible (depends on complexity and time):

Values below/above zero:
- line charts through zero
- area charts through zero, and flipping the area for negative values
- bar charts, flipped for negative values

Barchart stuff

The line supports gradients (vertically) to allow for hard and smooth color changes throughout the graph. It would be nice if the bar could also support this:

Well...... 😄

So:

bars with single color
bars with colorstop ('hard' gradient) applied to the background, where the bars themselves are a mask to reveal that background
bars with colorstop ('smooth' gradient) applied to the background, where the bars themselves are a mask to reveal that background
bars with colorstop applied to the bar, ie the full bar gets the color for the colorstop
bars with colorstop gradient applied from left to right. The value of the bars are not relevant. Just design.

Additional background:

The graph will only support ONE direction, that is horizontal. For other orientations, the SVG transformations rotate() and scale() are used to accomplish this:

rotate(90) to make a graph vertical
scale(x,y) with values 1, or -1 to 'flip' or mirror the chart in either x or y direction.

This only works for toolsets, but that should not be a problem.

This way any direction can be supported, without much work!

Related Issues (if any)

(Optional): Suggested Solution

Use existing parts, integrate and adapt parts of:

the graph part of the mini graph card
the existing bar chart
mix of styling and possibilities of the previous two

Thoughts about negative values

There can be a y_zero setting that controls the 'zero' y axis value. Default is 0.

the y position of that axis is important
height of bar is something like height = (.max - Math.max(y_zero, .min). If minimum value is -20 for instance, a bar with value 30 is now limited to 0 (assuming y_zero is 0) for its height, where without that restriction, the bar height is 50 (30 - -20).
a bar with negative value is drawn from y_zero to its height
a bar with positive value is drawn from height to y_zero

A line and dot chart is just drawn as it was before.

An area chart also needs to calculate the y_zero axis, as it should flip the area around this axis value.

A data test set must be used (fake data 'input') with fixed positive and negative values to be able to build and test this!

Difficult stuff...

The graph itself runs in the height/width of the SVG, which is the size of the Graph tool.

The line_width in both the x and y directions must be accounted for: the size of the graph must be adjusted to make the full lines fit into the SVG view. The main difficulty here is to adjust the background colorstop / gradient to exactly match the adjusted y values, otherwise, colors and the graph don't match.

So far, I failed to get background and foreground (the graph is a mask) to match exactly if the line_width is accounted for. If that one is 0, background and foreground match perfectly!!

left upper: perfect match
left lower: complete fail due to trying to account for line_width. Calcs are wrong!
middle part: perfect match
right upper: line is cut-off (no line_width compensation), but colors match perfectly
right lower: matching a line and a bar painted over each other (these are TWO graphs, not a single graph with 2 entities!)

The gradient must be calculated in a way that it accounts for half the line_width at the start and end of the y-axis colors.

The adjustments on the x-axis are fairly simple and don't have anything to do with colors. The total width is simply adjusted for the full line_width. That's it. As an extra, a margin-x can be specified to account for dots that otherwise are cut in half on the outer sides of the graph.

You have:

The background viewport. This is the full graph size and is used for the background color.
The foreground viewport. This is the smaller part of the graph itself. It accounts for the x and y margins and/or line_width stuff. I prefer a manual margin...

Things that don't work...

History fetcher

The current history fetcher for the bar tool can only fetch sensors / numbers, and can't deal with on/off or other textual states. This means the new graph tool is also limited by this fetcher. The graph tool has state mapping functionality, but that can't be used (yet).

If the history fetcher is rewritten, the graph tool could display more than sensors, but in that case it is unknown what the consequences are for the bar tool.

More than 1 entity in a graph

Not possible at this moment. The history fetcher can only push a single series of state to the graph tool (or any other tool), so multiple entities can't be done.

Furthermore, a tool can have multiple entities (entity_indexes field), but these are meant to be used in the animations section.

The field could be re-used, but supporting multiple entities is not very sparkline behavior, and specifying extra stuff per entity is also complicated. Remember that styling in that case should also allow for multiple lines, bar, etc.

So you would get:

  styles:
    0_line:
      <line styles for entity 0 in entity_indexes>
    1_bar:
      <bar styles for entity 1 in entity_indexes>

etc.

Animations

A graph doesn't have a single state, so there are no state operator animations possible.

This at least would make the entity_indexes field re-useable for more than one entity, as it does not clash with the animations.

Do add an extra field 'graph: true' to that section. If the list is used for other stuff, at least the graph can skip entities that are not meant to be graphed!!

AmoebeLabs commented 1 year ago

2023.06.16 Status

Just after a few hacks with a somewhat isolated Sparkline GraphTool... The basics seem to work. No integration yet, apart from being able to create the tool and render it. As I only want a sparkline, only the graph is re-used so far.

Some bugs are present, such as no working color stop for the bar, but working for the line graph. Weird. No idea why.

The next steps in the coming weeks are to integrate this, and make vertical versions possible, although I'm not sure if anyone is using the vertical display with the current BarChart...

Another hack with rotate/scale shows that a vertical graph (with dots in this case, another hack) can be made:

AmoebeLabs commented 1 year ago

2023.06.18 Status

Just a continuation of some hacks to discover if bars can be styled the same way as the lines/areas. So actually extending the bar styling with new options. Bars now work with background and masks. There is no visible difference between the previous version and this one, so that is nice.

This means bars can also show gradients with hard and smooth color changes, just as the lines can do.

The below example shows a 'hard' transition on the color stops, but each bar now has a hard-coded transparency gradient (again using a mask), so each bar only shows a part of the length. If this will be possible using 'styles' remains to be seen.

Hard color stop:

Hard color stop gradient with fade option enabled (same as for the line):

Smooth color stop gradient. Sometimes difficult to see, depending on the color stops used:

Smooth color stop gradient with fading enabled. Even more difficult to see the color stops used:

AmoebeLabs commented 1 year ago

2023.06.21 Status

Hacking zero crossings :smile:

I'm bad at math, so this was difficult. Some things work, but others have offset / line_width problems that are not yet calculated correctly.

If you take the third card and compare the bar chart with the line chart, you see the line chart has an offset. This is caused by the line_width.

But still, baby steps and progress!

It is a lot of work, and styling is yet to be implemented. All the opacity/gradient stuff is still hard-coded. I hope this can become user styled, instead of a configurable setting with fixed opacity/gradient settings.

Due to the zero crossings, calculations are needed to start/stop masks, gradients and fills at the zero crossing, so using a user-supplied style won't work unless I can think of a solution...

Addendum

And to be complete: the dark side of the sparkline graph. Some of the graphs just look nicer!

The upper left really looks like the Dutch Mountains ⛰️
The bar charts now show a gradient with 45 degrees rotation, from upper left to lower right instead of the usual top to bottom.

AmoebeLabs commented 1 year ago

2023.07.09 Status

I wouldn't say I like math :smile:

But still, some progress with traffic lights alike sparkline graphs...

On the left, is an AQI graph, in the middle is a 3x Awair Element temperature graph, and on the right 3x Awair Element humidity history graph. The Awair graphs are from top to bottom: study, bedroom, living room. So yes it's a bit warm in the study!

That's what you get if it is nearly 34 degrees Celsius combined with thunderstorms: hot & moist...

The VOC, CO2, and PM2.5 levels are all green, so nothing special to see on those sensors.

The previous circles are actually squares with rounded edges. So disabling them gives you rectangles. And in the lower right graph, squares are disabled: so you get real rectangles...

And just a little bit later: one peak in the living room... That is what you get while averaging the previous hours: the average peak might be different every 5 minutes...

The fifth Awair Element color is red btw. So I got almost all possible colors in 1 graph!

The circles still look better...

And if you take the lower right graph, and take a traditional approach using values, instead of buckets as is used for the traffic lights, you get an equalizer-like display (middle graph).

The middle graph and the previous first graph should have used the same colors, but as you can see, one has more yellow peaks than the other. So one of them is wrong :smile:. This can be color calculations or (middle graph) some gradient background that has gone wrong as the middle graph is using background with masking, whereas the traffic light graph is using direct colors for the rectangles/circles.

A dot graph (right graph) is showing two peaks of yellow over the last 24h...

Home Assistant history shows values over 51, so it should be yellow...

However, if I specify 48 buckets for the y-axis instead of 40, yellow it is. So it seems like some rounding and/or matching buckets with the background which might never match exactly...

Using 13 vertical buckets, you can see that a bucket might cover more than 1 color, which is logical, as a bucket is just a number, and is not aligned with the color stops making the background color.

A fixed lower/upper bound of the y-axis and a bucket count matching the boundaries should make it 100% work. But with a variable min/max y-axis and fixed number of buckets, this might or just might not work

Oops

I nearly forgot to include the min/max background graphs to be used for average graphs. Ain't they look nice (right graph top/bottom) 🥳

Data is from my DSMR reader, ie its electricity with 2kW peaks from my kitchen boiler.

Addendum...

Using the Awair color steps:

Note that the current implementation with color stops/thresholds can not implement the full functionality Awair offers, as Awair is using multiple buckets per color for temperature and humidity...

Instead of color stops/thresholds, color buckets should be used with multiple value ranges. The buckets should be in order and contain the color and value ranges

  color_buckets:
    - bucket: 1
       color: #xyz
       ranges:
          - range: 0..10
          - range: -20..-30
          -

Or, by re-using the color stops, add order based on color, and let the software re-arrange stuff to make those multi-range buckets work!

  color_buckets:
    - bucket:
        color: #xyz
     - bucket:
          color: #kkjd

Or add extra info in the color stop. Example for temperature. If colors are using a color swatch, the colors will even adjust depending on dark/light theme mode...

          - value: -1000
            color: '#e63740'
            bucket: 5
          - value:  9
            color: '#fdd125'
            bucket: 4
          - value: 11
            color: '#faaa00'
            bucket: 3
          - value: 17
            color: '#fdd125'
            bucket: 2
          - value: 18
            color: '#49ce4c'
            bucket: 1
          - value: 25
            color: '#fdd125'
            bucket: 2
          - value: 26
            color: '#faaa00'
            bucket: 3
          - value: 31
            color: '#fb8600'
            bucket: 4
          - value: 33
            color: '#e63740'
            bucket: 5

AmoebeLabs commented 12 months ago

2023.07.14 Status

Some clock stuff. Should maybe add a clock face too 😄

The left shows min/ave/max energy usage from today in 15 minute blocks. Middle is line graph of same sensor, and right one is showing 24 hour history with hour grouping… it should be identical to left ave ring, but has some bugs with the starting hour…

AmoebeLabs commented 11 months ago

2023.07.17 Status

Added a simple clock face (absolute and relative) for a clock day of 24 hours...

night / day circle (can be changed using CSS)
hour marks (can be changed using CSS)
hour numbers (fixed 4)

Since I liked the circle, I thought, why not make a sort of timeline:

electricity on the left
middle with occupancy and 2x pollen (trees/grass)
right same as middle but wider

It was nearly 15 minutes of coding because I could re-use a lot of logic 🚀

Entity state mapping is also working :smile:

And why not make a variant: audio for the last one:

Nice that the pollen trees vary a lot here (not for my lungs...), so I get nice colored graphs!

And while I'm at it: a variant: sunburst for the clock. Very similar to the timeline 'audio' variant, you can see the size of the value next to the color:

More sunbursting with the occupancy and the pollen trees and grass sensors.

This tool never gets finished 😥 . It's already 900+1600=2.500 lines...

AmoebeLabs commented 11 months ago

2023.07.18 Status

Trying to get logarithmic scales working for some of the graphs... This requires at least values of > 1. So temporarily had to multiply the energy values by 1000. I should use the state converters, but that is just too much work now.

The value of the lower_bound also matters a lot in how the graph is displayed. The default is auto min/max. This causes the influence of log10 to seem less.

first one uses log10 with a low lower bound of 0 to "compress" the chart
both the sunburst and the line graphs use log10. Sunbursts also use a low lower bound

Same first row, but now the timeline uses auto min/max, so NO lower bound setting. You can see that the difference between the values is again maximized. Thanks to log10, the first graph is still more compressed, than without (second graph).

AmoebeLabs commented 11 months ago

2023.07.19 Status

Some refactoring of the configuration, log10 scale, and working gradient calculations on the timeline and clock graphs...

Also added the ability to view 'yesterday'... as you can see below.

The time graphs can also use a number of lines and format them. These can be used to make some sort of x axis display.

The rest of the graphs still work, despite all the changes.

Dark mode also still working!

Some functional card examples...

Made two fce templates based on the original pollen-all template. Made the tree/grass/weed pictures smaller, and added a clock/sunburst in the first example and a timeline/audio graph variant in the second.