blog/kpi-evolution-decomposition/

[utterances-bot]

Answering "Why did the KPI change?" using decomposition • Max Halford

Motivation Say you’re a data analyst at a company. You’ve built a dashboard with several KPIs. You’re happy because it took you a couple of days of hard work. You even went the extra mile of writing unit tests. You share the dashboard on Slack with the relevant stakeholders, call it a day, and go grab a beer. A couple of weeks later, a stakeholder pings you on Slack, asking why some KPI changed.

https://maxhalford.github.io/blog/kpi-evolution-decomposition/

[martindaniel4]

Fantastic writeup Max! I hope it will be useful for your readers in their daily analytical work. Here are a few comments based on my experience at Airbnb:

• I found those frameworks useful when decomposing "growth rates" such as year over year values. The maths stay the same but the business question becomes what drove the acceleration between two points in time.
• You alluded to it in the MECE section, but of course you can use multiple dimensions to decompose. For instance if that made sense, you could combine new / returning visitors as well as claim type.
• A good combination was also to use anomaly detection across all Airbnb metrics. Once we identified a significant metric variation, we would decompose that metric using some dimensions to understand the root cause. My team and I had built a Python tool called "Mercury" to speed up that process but not sure it's still maintained internally. It's a great idea to open source it!
• To your point on semantic layer, that work was only possible because we had a strong metric / dimension foundation though the Minerva framework (https://medium.com/airbnb-engineering/how-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70). Folks like Robert Chang, Amit Pahwa, Vaughn Quoss were instrumental on that endeavor.
• I love waterfall charts to represent decompositions.
• Lastly I myself learned about those frameworks though Rodolfo Carboni who developed similar techniques at Amazon I believe. Alice Berthaux, Sara Liu on my team also did fantastic work on that front.

[BrunoGomesCoelho]

Wonderful read Max, thanks for sharing this!

The link for the visual metric tree isn't working, I'm curious to what the visualization was if you can share it!

[GStechschulte]

Great blog post and intuitive explanations. Good question regarding "I’m not sure what’s best between writing a Python package or a dbt extension". I would be interested in developing a Python package, but considering where and how most KPIs are computed in the "data stack", I begin to think the dbt extension makes most sense also.

[MaxHalford]

Thanks @BrunoGomesCoelho. I believe you have to create an account on the Count website to access that link. Here's a screenshot:

[aldbs]

Thank you Max for sharing this KPI decomposition technic ! I truly think the premise you have framed at the beginning of your article in one of the most important challenge that data analysts have to face with when interacting with business teams. Since I’ve read your article, I have been implementing this technic and it gave me pretty interesting results and triggered valuable conversations with business teams.

For ratio decomposition I found the geometrical analogy hard to understand and I have prefered a more « physics » oriented explaination starting from what you’ve said: KPI can evolves from 2 factors the inner effect and mix effect (based on your formula):

1 – Inner effect is how each category ratio evolution impacts the global ratio. Then for a given mix (hold constant between t and t+1) if all category ratios grow then your global ratio grows accordingly. And you compute how much in value each category contribute by multipling the difference by the share (at t or t+1 depending if you are performing a left or right decomposition) 2 – the mix effect : This can be more tricky to understand but a good way to see it, is: Let’s hold category ratios constant, then imagine you witnessed a raise in the mix on a given category, the impact on the global ratio depends on if the category ratio is higher or lower than the initial global ratio (because as you mentionned the global ratio is the weighted average). So if a category with a low category ratio increases in the mix then it impacts negatively the global ratio. And vice versa ; if a category with a low category ratio decreases in the mix then the global ratio increases.

FYI, I also like Martin views on how he uses anomaly detection to target what KPI to analyze.

Now I am focusing and working on how to create a systematic and robust way to perform this analysis by our business team, I’ll be happy to share views on the topic with you if interested.

[MaxHalford]

Hey there @aldbs. Thanks for giving your own interpretation of ratio decomposition. It took me quite some time to understand, and I have to admit my explanations were not great. I think it's wonderful that you're using it (at Cheerz if I'm not mistaken)!

I'd like to share that since this blog post, we've implemented funnel decomposition. The idea is to decompose metrics that can be expressed as products, such as:

revenue = impressions x click_rate x conversion_rate x average_revenue

I haven't yet written a blog post, but it turns out to be far more intuitive. Moreover, such formulas are more widespread in business units, so I'm hoping this kind of decomposition can be more useful. I've implemented/tested the formulas here with some toy data. Happy to discuss further.

[PoulpiFr]

There seem to be an error in the first decomposition graph, just under the coloured "B=A+C+D" formula in the Intuition paragraph.

I believe that it's showing a "left/right" decomposition instead of the advertised "right/left" =)

[MaxHalford]

Good catch @PoulpiFr! I just sent a fix. I'm glad you caught this mistake 🙏

[Vikram12301]

Hey Max, I have a doubt, In this statement, "The second bucket is due to the change in the number of claims. Let’s translate this image to an equation". Doesn't the change in the second bucket is also due to the change in the average amount per claim?