[SIP-113] Calendar Heatmap Improvement

[Mattc1221]

The problem

Calendar Heatmaps in Superset don't allow users to select the start and end of the heatmap range even though the calendar heatmap library that we use has this functionality. Documentation is here

The Motivation

Full Motivation

Data with specific thresholds

If your data has specific thresholds or boundaries that are meaningful for your analysis, manually setting the range allows you to highlight these thresholds accurately. For example, in a temperature heat map, you might want to distinguish between temperatures above and below freezing point with different colors.

Highlighting anomalies

When you want to focus on anomalies or outliers in your data, manually setting the range can help make these anomalies more apparent. For instance, in a sales data heat map, you might want to highlight exceptionally high or low sales figures.

Comparing different time periods

When comparing data across different time periods, setting a consistent range ensures that the color scale is uniform. This is especially important when comparing multiple calendar heat maps side by side.

Emphasizing specific values

To emphasize specific values or a particular range of values that are particularly relevant to your analysis, manually setting the range allows you to create a visualization that aligns with your objectives.

Ensuring consistent color representation

When working with multiple heat maps or sharing them with others, manually setting the range helps maintain consistency in color representations, making it easier for viewers to interpret the data.

Accommodating user preferences

Depending on your audience or user preferences, manually setting the range can allow for customization. Users may have different color perception or need to see the data in a specific way, and manual range setting provides flexibility.

Sensitivity to small changes

In certain scientific or engineering applications, you might want to create heat maps that are sensitive to even small changes in data values. Manually setting the range allows for fine-grained control to detect subtle variations.

Clarity and readability

For educational or presentation purposes, manually setting the range can enhance the clarity and readability of the heat map, making it easier for the audience to understand and interpret the information.

Highlighting specific events or patterns

Manually setting the range can be used to emphasize specific events or patterns in the data. For example, in a calendar heat map showing website traffic, you might want to highlight days with significant spikes in traffic.

Adherence to domain-specific standards

In some domains, there may be established standards or conventions for color-coding data in heat maps. Manually setting the range allows you to adhere to these standards accurately.

TLDR

An example of why we would want to specify the calendar heatmap range:

1. I want to display 2 different calendar heatmaps that span 2 different time spans.
2. I want to see the differences between the two time spans.
3. With the current implemtation we would get something like this:
4. These 2 maps look similar and at first glance convey that they are displaying the same data. However, the color range on these 2 charts don't match and can thus easily lead the user to bad data analaysis

Proposed Solution

1. Add a required chart control with the label Metric Range
2. The options for Metric Range would be Automatic, Manual
   • Note: the default value would be Automatic and use the same strategy we use now
3. Selecting Manual would display 2 new inputs: Metric Range Start and Metric Range End that would be integer inputs

https://github.com/apache/superset/assets/36670322/af22eda0-aab7-4b3d-8e98-57166966db68

[rusackas]

Thanks for the SIP (and the PR!). We're holding the PR while the SIP settles out. The next step here is to start a [DISCUSS] thread on the mailing list. Let me know if you want any assistance with that, you can just DM me on slack.

[rusackas]

One faced of the discussion here would be about the features we add to Superset's Calendar Heatmap. I don't have any push-back on any of your proposal in that regard.

I think the biggest fork in the road we'll have to contend with is which plugin we want to support going forward. I have no problem updating the current "legacy" plugin, but we would need to make moves to remove it's "legacy" status - namely upgrading to the newer chart data endpoint, supporting additional features (drill by, cross filters, etc) and sprucing it up overall.

Alternatively, we could go forward with an ECharts-based calendar heatmap plugin. That might be smaller/lighter/maintainable, but it probably won't be able to reach feature parity with what you're proposing here.

I'm personally OK with either direction, but nervous about adding more features to a legacy plugin without giving it the proper evolution/upgrades it needs to survive the next major version(s) of Superset.

[michael-s-molina]

Alternatively, we could go forward with an ECharts-based calendar heatmap plugin. That might be smaller/lighter/maintainable, but it probably won't be able to reach feature parity with what you're proposing here.

I would vote for moving to the ECharts plugin unless there's a really good reason that blocks us from moving into that direction. I even have a Heatmap PR (not calendar) that will land soon and may help with the calendar version. We need to think about UI/UX consistency and code complexity and try to always move to ECharts to materialize SIP-50.

[JohnDietrich-Pepper]

@rusackas @michael-s-molina

Right now, the calendar heatmap in Superset has some severe limitations, particularly around consistency when multiple heatmaps are on the same dashboard that this SIP would address. We can wait until the ECharts one is implemented but that could literally be a year or more away whereas this and the corresponding pull request could address that issue in months or less making the platform significantly more robust.

I realize you really want to move everything to ECharts but I personally think feature enhancements that allow Superset to be a viable alternative to other business intelligence platforms should come before trying to get everything on one Javascript library. ECharts has some good features but it also has some areas that aren't as evolved as controls like the one calendar heatmap is based on. I would vote for implementing the best library for a particular visualization.

[rusackas]

I realize you really want to move everything to ECharts

Not at all the case... We prefer using it when it makes sense, but there are plenty of plugins where it might not be the best choice.

I would vote for implementing the best library for a particular visualization.

100% agreed.

The main worry in my book is extending the features of a legacy plugin that doesn't use the current data endpoint or provide other first-class features like drilling. I just hope that gets prioritized sometime before the legacy endpoint is deprecated.

[michael-s-molina]

I would vote for implementing the best library for a particular visualization.

Any technical decision is always a trade-off. Any time you introduce a new library you need to consider all base logic that needs to be supported consistently throughout the app. Think about theming, tooltip handling, legend interaction, zooming, data transformation, etc. If you introduce a new library you need to implement these behaviors in a consistent manner with the rest of the plugins to avoid decreasing UX. At times, you want to introduce a new library because it provides many important features that are not supported by the common library and the cost of implementing the base logic pays off. I generally use the principle where if you can provide the plugin in a satisfactory manner using ECharts, that should be the preferred way. Only when this is not an option, I would introduce a new library.

[JohnDietrich-Pepper]

What are the next steps on this? We have dashboards deployed for customers that could leverage the work that Mattc1221 has already completed and would like to be able to use his updates without trying to fork and maintain concurrency with the official releases.

[rusackas]

Re-numbering this one to SIP-113, since another Issue was also numbered as 111, and went up for a DISCUSS thread on the official mailing list. Please proceed with opening a DISCUSS thread for this one when you want to move it forward.

[rusackas]

Discuss thread opened officially on the dev@ list :)

[rusackas]

@Mattc1221 I can carry this forward for a vote, if you're still hoping to implement it :)

[Mattc1221]

@rusackas that would be great! I would plan on just updating this PR

[rusackas]

Just got another piece of feedback that one of the major complaints with this chart is the orientation. It would be less confusing to users if there was a way to orient the calendar horizontally (like a wall calendar), or at least provide a control with the option for that sort of layout.

[rusackas]

I think this just needs one more binding vote to get passed. Please feel free to respond on the thread and call for help!

[rusackas]

Just pinged the thread on the Dev mailing list to try to get one more binding vote for this to pass. Anyone reading this (but especially PMC members) please feel free to vote!

[Mattc1221]

@rusackas how do you vote?

[rusackas]

Hi! To vote, you just reply to the Vote thread on the dev mailing list. If you're not on the list, you can join by emailing dev-subscribe@superset.apache.org. To see older messages, you can visit https://lists.apache.org/list.html?dev@superset.apache.org

[rusackas]

As it stands, there are 2 binding +1 votes (3 needed) and 2 non-binding +1 votes. I'll try to rally some folks for a third.

[rusackas]

Got the third vote! Closing the thread, the vote has PASSED.