Open barbeau opened 9 years ago
Proposal for simplifying the session reporting:
Reporting the OBA session discussed in https://github.com/OneBusAway/onebusaway-android/issues/237#issuecomment-77021844. Also, following table shows the currently reported session events when app starts.
Event Category | Event Action | Event Label | When? |
---|---|---|---|
app_settings | configured_region | API Region | App starts |
app_settings | general | Set Region Automatically | App starts |
app_settings | general | Show Experimental Regions | App starts |
app_settings | configured_region_auto | Set Region Automatically: %@; Old Region: %@ | Auto region config |
However, these reported events cause huge data in the Google analytics admin as shown in the image below:
Since, we are already reporting the API region as a secondary dimension, we don't need to report the API Region itself. Hence, we can simplify these events as shown in below:
Event Category | Event Action | Event Label | When? |
---|---|---|---|
app_settings | startup | Conf: [Set Region Automatically: YES or NO, Show Experimental Regions: YES or NO, ...] | App starts |
Furthermore, application preference change events logged under ui_action:
Event Category | Event Action | Event Label |
---|---|---|
ui_action | button_press | Turned on Experimental Regions |
ui_action | button_press | Turned off Experimental Regions |
ui_action | button_press | Set Region Automatically |
ui_action | button_press | Set region manually |
To better address and simplify these events, we might change the event category and rearrange these events as shown in below:
Event Category | Event Action | Event Label |
---|---|---|
app_settings | general | Set Experimental Regions: YES or NO |
app_settings | general | Set Region Automatically: YES or NO |
Thanks @cagryInside for the ideas.
If we drop region reporting at the startup of each session will we still be able to tell how many sessions each region had each day?
If we combine the current settings reporting I believe that will make it much harder to tell how many sessions are running with specific settings. For example, how would we build a report showing number of sessions today with experimental regions enabled.
Renaming the recording of set setting events will result in a lose of the current recording, is there enough benefit to warrent the change vs. just improving the documentation around what events mean what?
@bbodenmiller thanks, here is my comments:
If we drop region reporting at the startup of each session will we still be able to tell how many sessions each region had each day?
If we push single event, we still be able to tell how many sessions each region had each day:
Event Category | Event Action | Event Label | When? |
---|---|---|---|
app_settings | startup | Conf: [Set Region Automatically: YES or NO, Show Experimental Regions: YES or NO, ...] | App starts |
If we combine the current settings reporting I believe that will make it much harder to tell how many sessions are running with specific settings. For example, how would we build a report showing number of sessions today with experimental regions enabled.
I don't think it's so hard. For example,
First, choose the event category and event action in the GA admin:
Then, basically search whatever attribute you want to see in the GA admin:
Renaming the recording of set setting events will result in a lose of the current recording, is there enough benefit to warrent the change vs. just improving the documentation around what events mean what?
Yes, I agree that after these possible changes, we will lose the current recording. However, the current implementation looks like these in the GA console:
Since all set setting events logged under ui_event, its hard to distinguish which events are changing the app settings and which events are regular ui event. I though it would be better to have separate events for settings.
It looks like your proposal will work if we are willing to do manually data collection. I think either way we can drop configured_region
reporting since it is a custom dimension.
If we want to be able to use the Google Analytics Dashboards however I haven't found a way to make that work for the combined settings status event. Additionally the dashboard allows us to do some custom labeling so that, along with good documentation, could avoid confusion without resulting in loss of existing analytics data.
However we setup the analytics I think it'd be good to have some easy to use dashboards where we can pull relevant info similar to the following:
+1 for @cagryInside's proposal, and +1 for good dashboards
Additionally the dashboard allows us to do some custom labeling so that, along with good documentation, could avoid confusion without resulting in loss of existing analytics data.
To my understanding losing this existing data mostly isn't an issue, because the data already captured for iOS for the region changing "session" variables is already heavily polluted with bad data due to these events being reported in applicationWillResignActive
, which records every time the user starts the app, hits the home button, or gets a phone call (i.e., https://github.com/OneBusAway/onebusaway-iphone/issues/376). So the current data really isn't meaningful, and starting over with new labels is really the only way to make sense of it (other than always running reports for the data after which the above issue is fixed on iOS).
@cagryInside am I right in this regard?
To my understanding losing this existing data mostly isn't an issue, because the data already captured for iOS for the region changing "session" variables is already heavily polluted with bad data due to these events being reported in applicationWillResignActive, which records every time the user starts the app, hits the home button, or gets a phone call (i.e., OneBusAway/onebusaway-iphone#376). So the current data really isn't meaningful, and starting over with new labels is really the only way to make sense of it (other than always running reports for the data after which the above issue is fixed on iOS).
I think, existing data is not an issue with session events. However, there is a tradeoff between old and proposed configurations. Old one enables better dashboard usage (https://github.com/OneBusAway/onebusaway/issues/5#issuecomment-77326398) and the proposed version reduces the size of the reported data (https://github.com/OneBusAway/onebusaway/issues/5#issuecomment-77203278) and simplifies the event labels. PS: I also haven't found a way to make the proposed events work with better dashboards. Perhaps, there is a way to make it work.
From my understanding the only bad data is the region reporting which I agree we can remove since we have a custom dimension already with that data.
@cagryInside Can you please do a little more digging to see if you can set up custom reports, or another method, that could enable better dashboards? IMO not being able to create good dashboards would be the only thing blocking this proposal.
I've done some further research and have some suggestions. I believe that rather than track users current settings using events at app startup (which results in over counting, etc.) we should be using Custom Dimensions. This allows us to see the number of users running specific settings over a period of time, allows us to see which settings are most popular by region, allows us to see if users in experimental regions are getting the most crashes, etc. "Dimensions allow you to break down a metric by a particular value, like screen views by screen name. Dimensions correspond to the rows in a report."
Custom Dimension | Index | Values |
---|---|---|
API Region | 1 | API Region or "Custom URL" if custom API set |
Set Region Automatically | 2 | YES or NO |
Show Experimental Regions | 3 | YES or NO |
This change would result in a data loss of custom API counts and Experimental Region counts on iOS but it turns out we already are using a custom dimension for the API Region & Set Region Automatically setting so those counts would remain. Increased Contrast is another potential candidate to be a custom dimension. Note GA limits us to 20 custom dimensions.
I believe we should keep the following as is:
Event Category | Event Action | Event Label | When? |
---|---|---|---|
app_settings | startup | Startup View: view loaded at startup |
App start |
app_settings | configured_region_auto | Set Region Automatically: new region ; Old Region: old region |
Auto region config |
On iOS the Startup View (which is saved so the app always resumes to the same tab) is only recorded when the app starts fresh not every time the app is resumed and I'd recommend similar implementation on Android.
In regards to the other suggested changes around logging when a user changes their settings I believe we can address that through better documentation and dashboards rather than requiring a change in the logging structure and data loss. If you two as well as @aaronbrethorst & possibly @caitbonnar agree that the benefits outweigh the data loss then I'm fine with us changing it. Here is an example dashboard which I have setup on the tracker under Dashboards > Shared > Setting Changes:
I also added some details to and simplified the Tracking and Analytics page. It'd be helpful if someone could start populating the events section with a table of the exact events currently being logged by platform so we can identify where gaps exist or tweaks need to be made as a result of logging slightly different events (e.g. iOS and Android have same event name but log different actions). One could start putting that together by exporting data from GA using the following and the export to CSV option.
Can you please do a little more digging to see if you can set up custom reports, or another method, that could enable better dashboards? IMO not being able to create good dashboards would be the only thing blocking this proposal.
@barbeau We can show also the proposed config as following:
@barbeau @cagryInside any further comments on my proposed changes in https://github.com/OneBusAway/onebusaway/issues/5#issuecomment-77833275 or progress on tracking what events are tracked by platform?
/cc @aaronbrethorst
FYI @aaronbrethorst has added another custom dimension in the OBA iOS app per my suggestion in https://github.com/OneBusAway/onebusaway-iphone/pull/405.
Custom Dimension | Index | Values |
---|---|---|
VoiceOver | 4 | VoiceOver: ON or not set |
I might look in to changing it to the following just to make sure we know if any events are picked up without the dimension set. Update: PR at https://github.com/OneBusAway/onebusaway-iphone/pull/407
Custom Dimension | Index | Values |
---|---|---|
VoiceOver | 4 | ON or OFF |
If Android has a similar feature this custom dimension can be renamed (in the GA dashboard without an app update) then reused for Android.
Current design of OBA iOS and Android analytics is discussed here: https://github.com/OneBusAway/onebusaway/wiki/Tracking-and-Analytics
@cagryInside will post a proposal here to simply the "session" analytics that are logged when the apps start up, as discussed in https://github.com/OneBusAway/onebusaway-android/issues/237#issuecomment-77021844.
Tagging @bbodenmiller and @aaronbrethorst so they can be part of the discussion.