pymc-labs / pymc-marketing

Bayesian marketing toolbox in PyMC. Media Mix (MMM), customer lifetime value (CLV), buy-till-you-die (BTYD) models and more.
https://www.pymc-marketing.io/
Apache License 2.0
715 stars 201 forks source link

MMM end to end notebook plot is mislabeled/incorrectly normalized #1176

Closed cluhmann closed 1 week ago

cluhmann commented 3 weeks ago

In the budget optimization section there is this bit of code:

num_periods = X_test[date_column].shape[0]

budget = X_test[channel_columns].sum(axis=0).sum() / (
    num_periods * len(channel_columns)
)

fig, ax = plt.subplots()
(
    X_test[channel_columns]
    .sum(axis=0)
    .div(num_periods * len(channel_columns))
    .sort_index(ascending=False)
    .plot.barh(ax=ax)
)
ax.set(title=f"Weekly Average Total Spend per Channel (Next {num_periods} periods)");

This new allocation is then compared to the existing allocation like this:

fig, ax = plt.subplots(figsize=(10, 6))
X_train[channel_columns].mean(axis=0).to_frame(name="train").join(
    other=X_test[channel_columns].mean(axis=0).to_frame(name="test"),
    how="inner",
).sort_index(ascending=False).plot.barh(ax=ax)
ax.set(title="Weekly Average Total Spend per Channel", xlabel="spend");

But this latter code doesn't appear to normalize by the number of weeks and the per-channel spends do not seem to match across the 2 plots. So I assume that the second plot is either mislabeled (it's total spend rather than weekly average spend) or the latter plot is supposed to be normalized like the first plot is.

@juanitorduz

cluhmann commented 3 weeks ago

Also, some of these allocation plots are sorted but some are not. This makes it very difficult to compare across plots.

And it is not always clear what allocations are being compared in a given plot. There are at least 3 allocations. The original allocation in the observed data, the allocation after optimization, and the allocation after optimization with the budgetary constraints. Having the legends be more specific would help. So would some additional narrative after the plots to explain how the plot is supposed to be interpreted.

juanitorduz commented 3 weeks ago

Thanks for the description and reporting this! I will take a look! 💪