search

mozilla / telemetry-dashboard

Web-frontend for aggregated telemetry data
telemetry.mozilla.org
Other
89 stars 113 forks source link

Histogram display should show total number of telemetry pings received for the selected product/dates/etc #628

Open jfkthame opened 5 years ago

jfkthame commented 5 years ago

If I'm understanding the telemetry dashboard display correctly, when I view a histogram for a particular probe, the "Ping Count" shown in the legend to the right reports the number of telemetry pings that included the data item in question.

However, it does not (AFAICS) show me the total number of telemetry pings received for the relevant product(s) and period, if (as is common) some of those pings did not include this data item at all.

This makes it difficult to fully assess the significance of the counts shown in the histogram, because it's not clear what proportion of Firefox users (or sessions) are represented by the data. I think it would be helpful if the report showed the total number of pings received for the same period, alongside the number of pings that included the given data item. Would this be a possible enhancement?

(As a proxy for this, I've been comparing against the Ping Count shown for a telemetry item that I believe is always logged, such as DISPLAY_SCALING, but this is a bit awkward and I don't know if it's really a reliable workaround.)

chutten commented 5 years ago

That is 100% correct and is as designed. Sorry about that. Unfortunately your use case of "Compare this metric relative to the population that could have sent a value for the metric" is not one that's well supported by the Measurement Dashboard (MD).

@robhudson for whether the Measurement Dashboard v2 (mdv2) project will address this use case better, and to consider adding this to the list of "Things that mdv2 can consider doing better than MD"

But, rolling back a few steps here, the generic response I tend to give for MD's failings is "This means MD isn't the tool for the job". Maybe if we look at the questions you hope to answer with this data I can point you to a tool that'll better serve your needs?

Maybe find me on IRC or Slack for a conversation (or we can continue here, in glorious Github, if you'd like. But I think synchronous communication might be better suited) and we can sort this out?

hamilton commented 5 years ago

@chutten we've had conversations w/ more than one engineer who brought up a common pain point about MD – "how do I figure out the effective audience size for the filters selected?" And is (at least for me) a hard requirement for some version of mdv2 (maybe not mvp, but hopefully).