Major change in stats percentages

[JJdeGroot]

Describe the bug hampus_sethfors on Slack:

I just noticed that the iOS dark mode stats on appt.org has dropped from 35% a couple of months ago, to 18% now. Feels strange, anyone know what’s happened?

To reproduce Visit https://appt.org/en/stats/dark-mode

Expected behavior The stats shouldn't change this much.

Screenshots If applicable, provide screenshots, they help us understand the bug.

System Any browser.

Additional context Aug 23 is when the drop happened to 17.59% On Aug 22 it was 35.17%

{"name":"Ios - Preference ui style - Dark","feature_key":"preference_ui_style","property_key":"dark","platform":"ios","amount":1597286,"percentage":17.59}

vs

{"name":"Ios - Preference ui style - Dark","feature_key":"preference_ui_style","property_key":"dark","platform":"ios","amount":1597105,"percentage":35.17}

The percentage seems to have halved: 35.17 / 2 = 17,585 which would round to 17.59

Other stats seem to be unaffected based on 1 check of bold text

iOS bold text on Aug 23

{"name":"Ios - Accessibility is bold text enabled - True","feature_key":"accessibility_is_bold_text_enabled","property_key":"true","platform":"ios","amount":632002,"percentage":9.92}

iOS bold text on Aug 22

{"name":"Ios - Accessibility is bold text enabled - True","feature_key":"accessibility_is_bold_text_enabled","property_key":"true","platform":"ios","amount":631864,"percentage":9.92}

If 631864 is 9.92% then total amount of datapoints is 6.369.597 If dark mode has 1597286 then percentage of 6,3M = 25% That doesn't match 17,59 nor 35,17 :thinking_face: And if 1597286 is 35,17% then that gives 4.541.615 datapoints, with 17.59% it's double so around 9M

When investigating this bug, I would like to request a new feature of the total amount that's used for division to get the percentage to be included in the JSON files. The addition of this field won't break any of the current appt.org code.

[jorisbr]

Hi! Johan requested for me to look into this! Thanks for the detailed reporting. That's a helpful start! It would appear that one of the organisations didn't propagate any occurrence of the dark value for the field Preference_UI_style in the past. Since 21-08-2024 we're starting to see this data coming through which explains why we're seeing the stat for preference_ui_style change. The reason why this is changing so much is because this value competes with a lot of occurrences of the light value from the past.

In our real world example the data we were first seeing was contributed to only by org 1 and thus the percentage would be ~35%. Since the addition of data from org 2 this percentage is effectively being halved since it started with such a low amount of data. This amount is slowly increasing as the update with support for dark mode is becoming more common. The number of dark mode users is still low though on the side of org 2. I believe this is because the current source for this data is the iOS app settings instead of the iOS general accessibility settings.

This table contains all usage of the dark value both in absolute terms and as a fraction of the total usage of the Preference_UI_style field for the org that just worked on this. It shows that the fraction has increased since the 21st of August but since the total occurrences of the dark value is still very small in historic context we're not seeing this having an effect on the total fraction.

So three main takeaways:

• The dark value was recently implemented and this is why we see the sudden drop
• The preference ui style field is likely not implemented correctly by the same org that just implemented the dark field
• Since the dark value is recently implemented it has a long way to go to impact the historic statistics. This can be considered a weakness of the chosen aggregation. We've previously discussed a rolling window of a year to clean up older data, which would alleviate this issue somewhat.

I'm going to discuss with Johan what the most suitable solution is both short term and long term. I'll also discuss the proposed feature to add as a feature: the total amount that's used for division

[JJdeGroot]

Thanks for the update @jorisbr ! I think the biggest problem is the usage of app settings instead of system settings. For these statistics, we want to track global user settings and not app specific user settings. I think the new org should change the source of their dark mode % to the Android/iOS system setting field.

Plus the other things you mentioned would also increase the accuracy of the stats. Thanks and have a nice weekend :)