[FEATURE] Update Illinois app event logs

[yoonlees]

Is your feature request related to a problem? Please describe. There is no problem with the current Illinois logging events, but we want to enrich more.

Describe the solution you'd like Cameron need more data to the Splunk by adding the parent field for the Illinois App event logs, where the parent field would list what main talent/feature each action is taking place within.

Describe alternatives you've considered None

Additional context None

[mihail-varbanov]

Hi @yoonlees, it is not quite clear what you exactly expect the Illinois app to do:

1. What is exactly a "main talent/feature each action"? Could you provide an example of an existing analytics event and a value of "main talent/feature action" that you want added there, as well as the context (use case) when this event will get raised?

2. Can you define all possible "talents/features" that are expected the application to refer?

3. Is it sure there is no way to detect the "main talent/feature action" based on the current page, for example, or other event fields provided by the application?

4. What about the analytics events that are not related to any functional area of the application, i.e. we cannot associate a "talent/feature action" to them, like lifecycle events or HTTP request responses? Is it expected that all analytics events will have this new field, or just for some of them?

5. What is exactly a "parent field for the Illinois App event logs"? Could you point its location in event JSON structure, e.g. here:

   
   {
   "event":{
   "name":"livecycle",
   "livecycle_event":"background",
   "page":"RootPanel"
   },
   "timestamp":"2024-04-08T07:50:47.726819Z",
   "app_id":"edu.illinois.rokwire.ios",
   "app_version":"5.2.4+2004",
   "os_name":"ios",
   "os_version":"17.3.1",
   "locale":"en_US",
   "system_locale":true,
   "selected_locale":null,
   "device_model":"iPhone",
   "connection":"wifi",
   "location_services":"allowed",
   "notification_services":"enabled",
   "location":{
   "latitude":43.55160895270139,
   "longitude":26.387155148408578,
   "timestamp":1712562502798
   },
   "session_uuid":"604ef080-f57c-11ee-9ac2-b7b44019165a",
   "user_uuid":"c20a9811-038a-11ee-9b71-0a58a9feac02",
   "user_privacy_level":5,
   "user_roles":[
   "student",
   "fan"
   ],
   "accessibility":false,
   "icard_role":"",
   "icard_student_level":"",
   "student_term_code":null,
   "student_type":null,
   "student_type_code":null,
   "student_admitted_term":null,
   "student_college_name":null,
   "student_department_name":null,
   "student_level_code":null,
   "student_level_description":null,
   "student_classification":null,
   "student_first_year":null
   }

[yoonlees]

I will communicate with Cameron using your input and provide comments.

[cschwin2]

Hi @mihail-varbanov, thank you for the questions. I will probably answer your questions out of order.

At a very high-level, when we try to answer questions like, "How are users using the Illinois App?", "What features are they using the most/least?", etc., we typically try to look at the event.target values, sometimes using other fields like event.page to help identify what "feature" of the app a particular logged action happened within. However, it's often difficult to identify what "feature" a target might be associated with. Ideally, we would have a field identifying the main feature (discussed further down) that each event.target value "belongs to" for every log that has an event.target value, so we could quickly, easily, and accurately identify feature usage (which would significantly improve numerous areas of our analyses).

Here are some examples of the challenges we are facing:

In these, the event.page is constant (ExplorePanel). We can guess that the first log has to do with the "Dining" feature, the second with "Athletics", and the third with "Wellness".

(Dining)

(Ahtletics)

(Wellness)

The issue with the above example is that we have to rely on the event.target values and manually categorize each of them into their respective features. There are thousands upon thousands of event.target values in the logs, so this approach is not feasible. We tried to be creative with looking at multiple fields and values for identifying main features, but even a "basic" one started getting too messy with the logic and started leading to inaccurate categorization:

I believe that explains why the answer is "no" for: 3. Is there any way to detect the "main talent/feature action" based on the current page, for example, or other event fields provided by the application?

So, we are essentially hoping there could be a way that every target within a given feature could be categorized into what we have referred to as a "parent field", but "feature" would be the most straightforward. For example, if a user goes to Browse > Academics and then any of the pages, links, etc. within Academics, those logs could have something like "feature: Academics", anything tapped within Browse > Dining could have logs saying "feature: Dining", and so on. My understanding is that we would want "feature: x" added to all logs that have event.target values, indicating the user tapped something, with the "x" categorizing the log into the feature that the target is associated with. No need to worry about logs that don't have event.target values. Hopefully that answers: 4. What about the analytics events that are not related to any functional area of the application, i.e. we cannot associate a "talent/feature action" to them, like lifecycle events or HTTP request responses? Is it expected that all analytics events will have this new field, or just for some of them?

As for 2. Can you define all possible "talents/features" that are expected the application to refer?, I think the following would be a great start:

• All App Sections from Browse (on the bottom bar)

  • For the most part, section title could be the value for "feature: value"
    • Academics, App Help, Appointments, Big 10 Athletics, Campus Guide, Dining, Events, Feeds, Groups, MTD Buses, Research at Illinois, Wellness
    • For Wallet, we may want that one want broken up into its components, as something like "Wallet: Bus Pass", "Wallet: Illini Cash" (which would include the "Add Illini Cash", if possible), "Wallet: Illini ID", and "Wallet: Meal Plan"
    • For the Notifications, Polls, and Recent (all from Browse), I think the destination from any of those would likely fall within one of the parent fields/main features from above (e.g., tapping a notification that takes them to one of the groups they are in might be better to have "feature: Groups")

• For the other items on the bottom bar:

  • Academics and Wellness
    • These are already included as sections from Browse, so if going to Academics or Wellness on the bottom bar directly could just have the same name as the ones from Browse, that would be perfect.
  • Map
    • Anything logged with an event.target value when going to Map on the bottom bar would be good to have "feature: Map"
  • Favorites
    • Like the Notifications, Polls, and Recent (from Browse above), I think the destination from Favorites would likely fall within one of the parent fields/main features from above (e.g., maybe "My Dining Locations" is on my Favorites page, and going there should give "feature: Dining")

• I think that just leaves the Profile, Settings, and Notifications (bell icon) at the top

  • I would say anything within Profile or Settings could get "feature: Profile" and "feature: Settings", respectively
  • For the Notifications (bell icon), I have a similar line of thinking as the Notifications (from Browse), Polls, Recent, and Favorites above, where the destination would likely fall within one of the parent fields/main features from above (e.g., if they have a notification for a group they are in and they tap that notification to go to the group, I would expect to see "feature: Groups")

• Summary list of "parent fields" / "features"

  • Academics
  • App Help
  • Appointments
  • Big 10 Athletics
  • Campus Guide
  • Dining
  • Events
  • Feeds
  • Groups
  • MTD Buses
  • Research at Illinois
  • Wellness
  • Wallet: Bus Pass
  • Wallet: Illini Cash (including the "Add Illini Cash", if possible)
  • Wallet: Illini ID
  • Wallet: Meal Plan
  • Map
  • Profile
  • Settings
  • The targets from Notifications, Polls, Recents, and Favorites would likely be associated with one of the above features, so maybe having the destination feature be the value would be the way to go. If it would be a lot easier to just designate these four as their own feature, I'm sure we could find a way to further identify actions taken from these few "main features".

Finally, for 5. What is exactly a "parent field for the Illinois App event logs"? Could you point its location in event JSON structure, e.g. here:, I hope the first part has been answered (we have been using "main feature" and "parent field" interchangeably; they both just refer to a field that categorizes each event.target into one of the features from the list above). As for the second part of your question, it currently does not have a location in the event JSON structure, and is instead something we are hoping could be added. Something like this (I bolded the requested item):

{ "event":{ "name":"select", "page":"ExplorePanel", "source":"null", "target":"Caffeinator at Ikenberry Commons", "feature":"Dining" }

I apologize for the length of this, but I hope it helps clarify our request. Thank you very much!

[mihail-varbanov]

Hi @cschwin2,

Thank you for the detailed description. I think I got what you need.

I have just one question. When you talk about the new "feature" field that you want added for all "select" events (i.e. that have a "target" field defined), you associate two different semantics for that field:

A. If the user selects a dining location in a Map view, you expect the "feature" field to have a value of "dining", i.e. this is the feature of UI that we will present in response to that tap.

B. If the user selects any control that is part of given feature's UI (e.g. "any of the pages, links, etc. within Academics") you expect the "feature" field to have a value of that feature, i.e. this is the feature that is currently active, and in some way it is similar to the "page" field that we pass. For example, if the user selects the "Learn more about the 8 dimensions" button from Wellness / Today's Wellness Tip page, you probably expect to get "feature: Wellness".

I see some conflict between these two semantics. For example, in Map view, if the user selects the "All Events" target from the dropdown list at the top, what feature do you expect to get passed: "feature: Map" (i.e. we are currently in Map view) or "feature: Events" (we are exploring events)?

Another note that I have is that the "select" event is very low level. It is generally passed on all user taps. We display very different UI that not always can fall in these predefined features. A straight example is onboarding flow (for which we should probably add "feature: Onboarding"), but I am sure there will be cases where it would be difficult to associate a feature for given user taps. We will work this out later, most probably we will either leave "feature" field null, or define new feature entries for them.

@pmarkhennessy, there are hundreds of locations that we log user taps on, and each of them needs to get handled, so the work on this issue appears to be significantly large, something like a week or two (very roughly), FYI.

[cschwin2]

Hi @mihail-varbanov,

Thank you for bring up the two different semantics. I can see where that can get confusing. I think it would make more sense to try to follow the second one ("If the user selects any control that is part of given feature's UI..."). So, Map, Polls, Notifications, Favorites, and Recent (and anything else that redirects from one "main feature" to another) should have "feature: Map", "feature: Polls", etc. for the initial "select" event, and THEN selecting items within the new feature could be logged with the new feature value.

For example, let's say a user goes to Notifications and taps on a notification for one of their groups. While they are in Notifications, the log for selecting the group would get "feature: Notifications", but after they are taken to the group, anything they select in there would then get "feature: Groups". Similarly, while in Map, selecting a dining location would be logged as "feature: Map", but anything they select after be taken to the dining location could then get "feature: Dining".

Does that answer your questions about what we would expect to see in situations where an initial "select" event happens within one feature that takes them to another feature?

If that makes sense and would work, I think that may even be better than my initial comment, as we could then identify usage of Map, Polls, Notifications, Favorites, and Recent from the new "feature: X", and then we could use event.target to further explore how those features are being used. The updated list of "main features" would then be:

• Academics
• App Help
• Appointments
• Big 10 Athletics
• Campus Guide
• Dining
• Events
• Favorites
• Feeds
• Groups
• Map
• MTD Buses
• Notifications
• Polls
• Profile
• Recents
• Research at Illinois
• Settings
• Wallet: Bus Pass
• Wallet: Illini Cash (including the "Add Illini Cash", if possible)
• Wallet: Illini ID
• Wallet: Meal Plan
• Wellness

Any other features that you see fitting this request, such as "Onboarding", like you suggested, would be great to have. In the cases like you mentioned where be difficult to associate a feature for given user taps, I understand if they would need something like "feature: null" or have new feature entries defined for them.

Thank you very much for all this! Please let me know how else I can help.

[mihail-varbanov]

Hi @cschwin2,

Thanks, everything that you say makes sense.

I have just one question. If we use the semantic "B" for the new "feature" field, i.e. to indicate what is the currently active feature's UI when a given object is selected (tapped), doesn't this overlap pretty much with the existing "page" field? For example, "page: ExploreMapPanel" equals "feature: Map", "page: Event2HomePanel" equals "feature: Events", and so on.

I noticed that Dining UI reports "page: ExplorePanel", that is some inheritance from the past when we used to display mixed types of objects/locations in a single panel named ExplorePanel. This, and other cases like this, should be easily fixed to use some more appropriate page names, and would require significantly less development effort than introducing this new feature in hundreds of select loggers. Please think on this option and let me know what you think - is it a good workaround and do you see any problems or difficulties?

[cschwin2]

Thanks @mihail-varbanov,

I can do a little more digging to see if that approach has any problems. If we could make that work, that would be fine with me. I think the concerns about using event.page are:

1. I looked at logs for the last 24 hours alone and found over 100 distinct event.page values
2. Many of these event.page values are not obvious (to me at least) as to which feature they are associated with (I think due to inheritance you mentioned), such as "page: ExplorePanel" being associated with Dining, Athletics, Wellness, and I'm sure others.

If instead it would be much easier to have some kind of mapping between event.page and the more narrowed down list of main features from my previous message, that would be fantastic. I will also keep looking into that field to see if I can identify any other potential solutions, problems, or difficulties with using it.

Thanks again!

[mihail-varbanov]

@cschwin2, if you send me the collected distinct values of event.page, I can try to create the mapping between page names and features.

FYI, we are trying to use naming conventions that generally includes the feature name as a part in the page name. For example, Event2HomePanel, Event2DetailPanel, Event2FiltersPanel, etc. refer Events feature; SettingsHomeContentPanel, SettingsNotificationsPreferencesPanel, SettingsVideoTutorialPanel, etc. refer Settings feature, and so on.

If you could collect page values that are not clear which feature they belong to, we could estimate if this workaround can be easily adjusted to do what you need, or we will have to take the longer way of adding the new explicit "feature" field anyway.

[cschwin2]

@mihail-varbanov, thank you for that information. This might be a much easier approach, if we are able to use the page name rather than each target to get the main feature. I will work on putting together a list of distinct values of event.page, and will also note which ones are not absolutely clear to me which feature they belong to.

[cschwin2]

illinois_app_pages_features.csv

Hi @mihail-varbanov, I attached a CSV file with a list of all (180) distinct non-null event.page values since the start of this 2023-2024 academic year. I also categorized each of these into the features that I believed each of them to be associated with, or used "unsure" (58 event.page values) if I wasn't certain. Some have additional notes.

Some of the ones I was unsure about will likely just be clarified as to which of the features they belong, while some others, like "ExplorePanel", may be associated with multiple features and would need to be investigated a little bit more.

Please let me know if you have any questions. Thanks again!

[mihail-varbanov]

Thanks @cschwin2, you did a great job collecting all distinct page names. Now I am checking the entries that are not clear to which feature they belong to. This will take me some time because this is related to a good digging into source code.

[cschwin2]

Thank you @mihail-varbanov. I am also continuing to look into potential challenges, at least with some specific case, we might face with this approach. I hope to have some more insight soon.

[mihail-varbanov]

Hi @cschwin2,

I processed all entries from the spreadsheet that you created. I set the feature values where it was unsure, as well as corrected a few of the assumed features for the other entries. I also added comments to almost all entries that I hope should make the picture clearer. I am attaching my response as xlsx file in order to keep the formatting that I added for some convenience.

There are a few questions:

1. Polls or Events in Groups to which feature bellong. During the process of analyzing event.page values I get more and more convinced that we should consider them as part of Groups feature. My feeling is that you are on the same page as me.

2. We should decide how we would treat McKinley appointments - as Academics or as Wellness features, please see my comment for AppointmentDetailPanel. I am OK with any decision.

3. Container and Padding values for event.page are rather a bug. You should either ignore these values, or if you keep getting them please let me know for this and send me the log entries where they persist.

4. There are some lower level panels that are used in different features. Examples: WebPanel, ModalImagePanel, etc. I will add the feature name in the event.page value, e.g. WebPanel (Events) or WebPanel (Academics).

So to summarize, ​​I don't see any insurmountable problem with using event.page values for determining the source feature. I will have to make some updates in the app in order to handle this, that is a relatively quick job. Please review my answers and comments, and if you do not see any problem either I will start doing the necessary updates in the app.

illinois_app_pages_features.xlsx

[cschwin2]

Hi @mihail-varbanov,

I apologize for the delay in my response. I am working with some other folks to get their input on how they want some of these pages categorized. As soon as I have their feedback, I will update you.

Just to clarify, though, is the plan to update the event.page values and then use that field to add a feature field to the logs, or is the expectation that we would just need to query against the updated event.page values to extract the feature?

[mihail-varbanov]

Hi @cschwin2,

No problems about the delay, I have plenty of other stuff to do.

I originally intended to leave the {event.page : feature} mapping to you. Now I realize that it would be better to put this mapping in the client, and to pass an explicit feature value, based on this mapping. In case of some ambiguity I can define the right feature value that needs to get passed. I need to review this stuff again in depth, but for now my feeling is that we would go to passing an explicit "event.feature" parameter to all analytics events.

There are some unclear questions that we need to answer. They are marked with green color in the spreadsheet from my reply, which indicates that either it is ambiguous, or that I have something to do in the client app. I will consider them again, and will discuss with you what have left unclear.

[cschwin2]

Thank you @mihail-varbanov!

I sent the spreadsheet to some other folks to provide their input and asked them to especially focus on the green color. Do you want me to ask them to wait until you have further reviewed and identified anything left unclear, and then we can get some final input from them? Or would you prefer to wait until we hear their input before you review to see if anything is still unclear?

[mihail-varbanov]

Hi @cschwin2,

I think I am ready with the first version of analytics "feature" support in the Illinois app. The app supplies a new property event.feature to (almost) all analytics events, similarly to the existing property event.page. In the very beginning of the app startup there is no feature defined yet, so it can be null, but as soon as the app UI gets initialized and available, we start supplying the feature property. You can find the complete list of supported features here.

In addition, we supply "target_feature" to all "select" and "map_select" events in ExploreMapsPanel (the Maps tab). This was the initial use case you gave as an example of the need of the feature attribute.

@pmarkhennessy will supply you with a build on dev environment where this is available. Please test and check if it does what you need and what we agreed to implement in the Illinois app.

[cschwin2]

Hi @mihail-varbanov,

I apologize for the delayed response. I am still working through a little bit of testing of the new feature field. So far, it is looking VERY helpful, so I really appreciate all this work.

Just to clarify, should the feature field be null only upon startup, or will there be any other instances where it will be null?

Also, is the integration of this new field something that needs to be part of a new app version release?

[mihail-varbanov]

Hi @cschwin2, I am glad that you find this helpful.

Just to clarify, should the feature field be null only upon startup, or will there be any other instances where it will be null?

Theoretically the feature should almost never be null. It can be null if we log analytics events before we have presented the app's UI, i.e. on app launch. BTW, I defined the "onboarding" feature and it is applied during the onboarding.

Also, is the integration of this new field something that needs to be part of a new app version release?

I am not sure exactly what you are asking. This new field will persist in the upcoming 6.0 release.

[cschwin2]

Thank you @mihail-varbanov. I did see the "onboarding" feature and appreciate that. That's good to know about the null feature.

I apologize for being unclear with my question about this field being included in the update, but you answered it. I think we may have some minor adjustment requests for the feature classification, so I wasn't sure if those changes would need to wait until another release.

[cschwin2]

@mihail-varbanov, I have a question and a few requests:

Question:

• Is there somewhere I could see/export a list of of event.page values (or whichever field is being used) and their associated event.feature value? I am essentially trying to see if I could map logs from before the event.feature field was released to the appropriate feature. I'm just not sure exactly how an event.feature value is assigned.

Requests:

The Illinois App team has asked if we could have the following modifications to the event.feature field:

1. Breakdown the "Academics" event.feature value into the individual components like Appointments, Campus Reminders, etc. (like what was done for Wallet), with the highest priorities being My Courses and My Canvas Courses.
2. Breakdown the "Wellness" event.feature value into the individual components (like what was done for Wallet), with the highest priorities being Mental Health Resources, Wellness Resources, My Success Team, Illinois Health Screener, MyMcKinley Appointments, and I'm Struggling.
3. Feeds no longer exists as a feature; we now have Radio Stations and Illini News. If we could remove Feeds as an event.feature value and have the Radio Stations and Illini News values added, that would be great.
4. Change the "Browse" event.feature value to "Sections".

Thank you very much!

Cameron

[cschwin2]

@mihail-varbanov, in addition to the comments above, I also noticed that the event.target_feature field does not appear to be acting the way I would expect. I looked at the last 30 days for all "select" events (event.name="select") where the event.target was NOT "Back", "Close", or "Home", and it looks like the event.target_feature field is null for nearly all logs.

If a user was in, for example, the Dining feature, and they tap a target that is also within the Dining feature, I would expect event.target_feature to equal "Dining", but I apologize if I am misunderstanding the purpose of this field.

Thank you,

Cameron

[mihail-varbanov]

Hi @cschwin2,

Is there somewhere I could see/export a list of of event.page values…

Not at the moment. Creating such list is quite huge and laborious task that I cannot handle at the moment.

I think I did all your requests (1 - 4). They are available from version 6.0.45. The feature names do not follow exactly the strings that the app displays (as they get changed quite frequently). You can find the list of available feature names here.

Regarding the target_feature parameter, it is only sent from map. If you remember our initial discussion at the beginning, passing the target feature for all select events is huge and tedious work, and we agreed to pass only the feature parameter that indicates the currently active feature. I added target_feature anyways for Map only because this was the example that you gave me for the necessity of analytics features. Anyway, what I did now was to not include the "target_feature" param if it is null. Hope this helps, otherwise I will remove it completely until we are ready to handle this everywhere.

Finally, please create a new issue in our repo when you need some updates regarding features, etc. as this one became too long as a Scandinavian saga.

[mihail-varbanov]

@cschwin2, FYI, a little cleanup and polish was performed in 6.0.46.

[cschwin2]

@mihail-varbanov, thank you so much for everything you have done with this request. I believe it has significantly advanced our reporting capabilities.

Regarding the target_feature parameter, it is only sent from map...

That makes sense to me. I apologize for my confusion around that parameter. I completely understand the challenge and will likely have minimal use of it until it becomes handled everywhere. Please keep it available, though, as I would like to explore its use with Map.

Finally, please create a new issue in our repo when you need some updates regarding features, etc. as this one became too long as a Scandinavian saga.

I understand and will do so moving forward.

Thank you again for all of you hard work on implementing this request!