[Lens] Store raw values for color mapping assignments

nickofthyme commented 2 weeks ago

Describe the feature

I propose we leverage https://github.com/elastic/elastic-charts/issues/2527 to store the raw unknown values of the color mappings in the saved object. These values can then be retrieved and used to compare with values in the chart and also to render formatted values to the user in the color mapping assignments UI.

Describe a specific use case for the feature

When assigning color mappings, we aggregate all the possible categories defined in the visualization then convert them to a string value, not formatted value. This assumption is problematic in a few ways listed below but at the heart of the issue there are 2 concerns:

How do we accurately match assignments to their correct aesthetics in the chart?
How do we display the assignments to the user? Formatted or raw values?

Ranges

Stringifing a range object returns something like "from:0,to:100". The problem is that in charts we do not have a way to return the raw value because we expect the accessor to return a primitive value (i.e. string, number, etc.). This requires Lens to pass the Chart the formatted values (e.g. "0 -> 100") of the ranges. Thus lies the issue defined in https://github.com/elastic/kibana/issues/187519, as "from:0,to:100" !== "0 -> 100".

Notice the differing thousand separator, one using , the other .. This was created by selecting a color assignment, then switching the locale.

Multi-field aggregations

Similar to the range issue, here we capture the separated values and the formatted values passed to the chart may be different than the saved values.

Notice the missing , in the number which is preventing the correct color matching.

Values with special tokens

In kibana visualizations, we have a few special values that are treated differently than others, for example the string "__other__" is converted to "Other". The problem with only storing the stringified, is that we are unable to discern which "Other" value is the actual keyword and which is just the string "Other". I would argue this is a rare edge case but does have merit for other cases as outlined in https://github.com/elastic/kibana/issues/190174.

Formatting values

With the current approach of stringifing the raw values, not formatting, we are displaying raw values to the use which can negatively impact the usability of the assignment selector. I'm not aware of any case in kibana, outside of value inputs, where we show the user unformatted values.

Dates

Currently dates are shown as in Unix time (e.g. 1726515421) which is not for human consumption!

We disable color assignments for dates in Tables, is this desirable to have?

Ranges

Currently ranges are shown as the id of the RangeKey class (e.g. "from:0,to:1000"). But this can be confusing to the user as the values are not in the same format, nor does every user understand what undefined mean in the context of a range, see example below.