[SIP-80] Traversing Hierarchical Data ("Drill Down")

[EvilDrW]

[SIP-77] Proposal for Traversing Hierarchical Data ("Drill Down")

Motivation

Many data sets have a natural hierarchy. When viewing these data sets, a common user request is to apply filters for each tier of the hierarchy iteratively. While superset provides accommodations to define these filters, the available methods are far from intuitive. A more appropriate user interaction would be:

1. Display only values for the highest tier in the hierarchy
2. User clicks one of the displayed values
3. Superset filters based on the selected value
4. Return to 1, showing only values for the next tier

Proposed Change

The proposed feature will provide an "enable drill down" check box on the control panels for all compatible chart types. Examples of possible compatible charts include pie, bar, tree maps, etc. When selected, the feature will use the column names in the "group by" field, in the order that they appear, as the data hierarchy. The iteratively selected filters will, as usual, be applied across all charts in the current filter scope (tab, dashboard page, etc.). The current state of the "drill down" will be tracked in the chart's OwnState object, as necessary.

For a demonstration of the proposed change on a pie chart, see the videos on https://github.com/apache/superset/pull/14688

New or Changed Public Interfaces

The only change to the interfaces, as envisioned, is the addition of the "enable drill down" checkbox to appropriate charts' control panels.

New dependencies

n/a

Migration Plan and Compatibility

n/a

Rejected Alternatives

n/a

[villebro]

I would also love seeing drill down/through functionality added to Superset! I think there are roughly two ways forward here: 1) adding drill down in the chart layer, much like has been done in #14688 2) adding support for hierarchical dimensions to the semantic layer.

Both probably have their pros/cons:

• Adding drill down to a single chart is fairly straight forward and wouldn't require a lot of changes to the rest of the application. On the downside it's slightly hacky and might introduce some design debt.
• Adding semantics for defining hierarchical dimensions would make it possible to introduce drill downs into any chart (including native filters), but on the downside it's a fair bit of work and could take long to implement.

As we're getting ready to kick off work for Superset 2.0, I'd personally vote for the second option, as I feel having full drill down support would be a very valuable (and cool!) feature.

@betodealmeida , has any consideration been given to adding hierarchical dimension semantics to the new dataset models?

[geido]

First of all, thanks @EvilDrW and @zhaoyongjie for moving this forward. I believe this functionality is a must have and would love to see it in Superset.

However, I also feel we should go for the most scalable solution rather than implementing the drill down on a chart by chart basis.

Adding semantics for defining hierarchical dimensions would make it possible to introduce drill downs into any chart (including native filters), but on the downside it's a fair bit of work and could take long to implement.

As we're getting ready to kick off work for Superset 2.0, I'd personally vote for the second option, as I feel having full drill down support would be a very valuable (and cool!) feature.

I agree with what @villebro is proposing.

[Vygo]

@geido @villebro Thank you for your feedback. Is there an estimate of when this would do with the 2nd approach? Is the target release date for Superset 2.0?

@EvilDrW is an engineer here at Stylo. As a company that actively uses superset this functionality is crucial for us and I would love if we could get this in as an MVP, while the team is working on the most scalable/long-term solution.

[betodealmeida]

@betodealmeida , has any consideration been given to adding hierarchical dimension semantics to the new dataset models?

@villebro We have not, but we definitely should. It could be as simple as defining the parent column in the dataset, eg, in city we'd set state as the parent dimension, in state we'd set country, and so on.

@EvilDrW I have questions about additive metrics. When you're at the country level, how do you compute the metrics, do you run a query with only country in the group by or do you run the full group by (country, state, city) and add the values? If the latter we should think how we're going to handle non-additive metrics like COUNT DISTINCT — this is another area where the semantic layer can help.

[shadowDy]

When does superset plan to support drill down？ thanks~

[EvilDrW]

@villebro @geido @betodealmeida: thanks for the feedback. I'm sure that what you're requesting is the right way forward for the project, but it is quite far beyond both my interest and skill level. I'll be interested to see how this progresses, but it will have to be someone other than me who implements it.

[srinify]

Hey there, I renamed this to SIP-80 since there's already a SIP-77: https://github.com/apache/superset/issues/17296

[crazamahdi]

Is the drill down feature out? @EvilDrW I am not able to see enable drilldown feature in my superset can anyone please help me implementing that feature in superset (if the feature is out)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

[mdeshmu]

When is "Drill Down" feature planned to implemented in Superset Roadmap ? At present we are using a proprietary BI Tool that supports Drill Down. We want to migrate our dashboards to Apache Superset but not having Drill Down in Superset is a blocker. Appreciate any response.

[Sandpy12]

When is "Drill Down" feature planned to implemented in Superset Roadmap ?

[Sandy381]

+1 Waiting for drilldown feature!

[crazyisjen]

+1 waiting for drilldown and drill through feature!!

[rusackas]

Some initial drilling features are being developed in the normal course of feature work. This SIP may require substantive updates to keep current with that effort if you wish to keep it open. It will also need to be followed through with on the dev@ mailing list to close out the discussion and bring it to a vote. Thanks!

[crazyisjen]

initial drilling features-- > u mean sunburst chart? any more choice? thanks

[rusackas]

initial drilling features-- > u mean sunburst chart? any more choice? thanks

By this I mean that recently there has been the addition of Drill to Details, which many of the newer charts support. This is currently behind a feature flag, but allows you to see the underlying data that a chart represents.

The next planned feature is Drill By, where you can right-click a chart, and quickly add a filter and select an additional column to Group By. This will essentially allow drill-down (and more), but you need to know the columns/schema/hierarchy of the data.

Following that, adding hierarchical relationships between columns at the semantic layer should allow the Drill By feature to be extended into Drill Down/Up actions more easily.

[sknudsen]

@rusackas Thanks for bringing these news to us. This sounds like great news or am I missing something?

Am I getting it right, that some of the ideas proposed in this issue are already implemented or is on the roadmap to be implemented soon? If that is the case, I wonder if the work done/being done is inspired by the description in this issue, and if not, how come (this question is coming from someone who lack an understanding of the project org)?

Also, where would I be able to read more about the feature flag and the planned features? Where do these discussions and design ideas/developments live?

Thanks

[rusackas]

A lot of folks have wanted this feature for years, so there's been input/inspiration from numerous sources. A lot of conversation has been happening in various open fora, including the #drilling-actions channel on Slack, and in other small working groups that have spun out of Town Hall discussions. Thus far, Preset has done the bulk of the work, but other individuals and orgs are more than welcome to join these discussions, especially if there's an intent to contribute to the implementation effort.

The hierarchical mapping at the semantic layer is an un-tackled design problem thus far, which will be given more attention after those other precursor steps are production-ready. Drill to Detail is already available as mentioned, and we're fairly early in Drill By, but I'll ping @kasiazjc @lauderbaugh in case we have any Drill By design/docs we're ready to share out in this thread.

[lzfxxx]

@rusackas

Drill to Detail is already available as mentioned

Looks like DRILL_TO_DETAIL feature flags is available only in master branch and in the development phase, and the newest release 2.0.1 doesn't include this feature. So I'm wondering when will this feature be released?

[rusackas]

So I'm wondering when will this feature be released?

With whichever release is the next cut of master. That might be 2.1 or 3.0 depending on which way the wind blows. Hopefully by early spring... though it's largely a volunteer effort, so take that with a grain of salt ;)

[rusackas]

Hey all... I'm closing this feature for a few reasons: • It was never brought up for discussion or a vote • DRILL TO DETAIL has shipped, and is on by default as of Superset 3.0 • DRILL BY has shipped, and is on by default as of Superset 3.0

If ayone wants to continue discussing/proposing additional drilling features, I propose that we regroup on a new SIP and rekindle the process.

[ghitti]

who can share how to configure drill-by using "header table" and go down to "detail table" clicking on a dimension. thanks