UX Considerations for display of mode percentages

[JGreenlee]

Since the latest release, the app now shows modes and percentages on the map of each trip, color-coded by the mode of travel for that section.

The intent is for this to be useful to users as an indication of what OpenPATH has 'guessed' that their trip was composed of. Before I label these trips, I can see that the first one was detected as Car and the second one was detected as Walk. This may help me recall what I was doing that day and may aid in me labeling the trip.

We received some valuable feedback from a long-time user, T – he shed some light on how the way the information is presented may cause confusion.

Consideration 1

T did not have any recent multi-modal trips in his diary, so all of them showed up as "100%". He wasn't certain what this meant. He understood that the icons were guesses about the mode of travel, but he thought "100%" might be a confidence level or something like that. T said that if he saw a trip that said "50% car, 50% walk" he would immediately understand that it was referring to a trip where half of it was by car and half was walking. But because he didn't have any multimodal trips, seeing "100% car" or "100% walk" was ambiguous.

He also noted that some trips had a question mark with "100%". He could guess that this represented 'unknown' mode, ie. OpenPATH doesn't have a good guess for what the mode was. But in that case, if OpenPATH doesn't have a guess to give me, what's the point of even showing the question mark?

My takeaway from this:

• If the trip is unimodal, we don't need to show "100%" next to the icon. We can just show the one icon and let it speak for itself.
• If the trip is unimodal and the mode is unknown, we shouldn't even show the question mark icon. We will just show the trajectory in grey without any icon above it.

---

Consideration 2

What do we do about these percentages after the trip is labeled?

In the above screenshot, the guessed modes were correct – when I labeled those trips, they were congruent with what the map was showing. But what if the 'guess' is wrong? Consider these examples:




These trips were detected as "bike" and "walk" respectively. They were actually car and e-scooter and I labeled them as such. It seems wrong that the map still shows "bike 100%" and "walk 100%".

What makes this tricky is that these actually represent different things in the data model – the icons+percents shown above the map are the inferred sensed modes for each section of the trip, and the user-provided labels represent the primary mode for the trip as a whole.

So when a trip is labeled, what should we expect to see on the map? A few options I can think of: 1) We leave the percents how they are and somehow clarify that these are 'guesses from before labeling' -- I don't really think that this information provides much value to the user once they have labeled mode, particularly if the guess was wrong 2) The section modes just go away after the trip is labeled and we override with the labeled primary mode -- we would lose detail about multi-modal trips. Once labeled with primary mode, everything would appear unimodal 3) If the primary inferred mode doesn't match what was labeled, we override with the labeled mode. But if the guess does match the label, leave it as-is -- we would preserve detail about multi-modal trips, as long as the label was congruent -- but a trip that was detected as "75% car, 25% walk" and labeled as "Car", would still show "walk" on the map, even if the trip was 100% car in reality

[JGreenlee]

Seeking input on Consideration 2 if @shankari, @Abby-Wheelis, @rahulkulhalli and/or anyone would like to share thoughts

[shankari]

I have not heard the unimodal 100% confusion issue before, but I have heard the

It seems wrong that the map still shows "bike 100%" and "walk 100%".

feedback a lot back when the diary was the primary mode and we had no maps in the label

My 2 cents:

• We leave the percents how they are and somehow clarify that these are 'guesses from before labeling'

This would be cool and would be consistent with the current data model (where we store both in separate fields), but I don't see how you can easily do that with your current representation.

I have often thought that it would be cool to have some kind of explainability here - in the broad theme of https://en.wikipedia.org/wiki/Explainable_artificial_intelligence where there could be an expanded view that would show the sensed mode, the full set of inferred modes (now) or maybe the path in the tree (once we implement random forest, @humbleOldSage). When we first implemented the label assist, in particular, there was significant interest in how it worked from some participants and program admins. There is an NREL researcher working on cataloging explainable AI work in the lab who emails me periodically - happy to put you in touch with her.

So I would categorize this as potentially very cool but complex.

The section modes just go away after the trip is labeled and we override with the labeled primary mode

This is consistent with what happens now with the downstream analysis (primary mode is treated as applying to the full trip) and will happen after "count every trip" is implemented (use user label when it is present, fall back to inferred label/sensed mode if not present)

This is fairly straightforward and might be way to start in the short term 👍

If the primary inferred mode doesn't match what was labeled, we override with the labeled mode. But if the guess does match the label, leave it as-is

I am not sure this solves the problem - you will still see some level of discrepancy between the user label and the sensed values. And it will lead to confusion about how the data is used (do we use the user label or the sensed or ???), so I downvote this 👎

If/when we change how we use the data for downstream analyses, we can change this.

[JGreenlee]

The section modes just go away after the trip is labeled and we override with the labeled primary mode

This is consistent with what happens now with the downstream analysis (primary mode is treated as applying to the full trip) and will happen after "count every trip" is implemented (use user label when it is present, fall back to inferred label/sensed mode if not present)

This is fairly straightforward and might be way to start in the short term 👍

I see. So once the trip is labeled, the mode indicator above the map can reflect the labeled mode. And we will display the trip on the map as unimodal from that point onward.

In that case, I think it's fairly important that users can still view the "detected modes" on the details page so that this information is not completely hidden from them once they label the trips. I am not ready to implement a full breakdown of sensed -> inferred -> confirmed at this point in time, but a simple toggle between "detected" and "labeled" can be done now without too much additional work and without requiring server changes.

It would be cool to use "segmented buttons" to toggle between "Detected" / "Labeled", similar to "Overall" / "Daily" in this example:

[JGreenlee]

But to even do this at all under the existing architecture, we need a mapping of what 'Mode' label choices correspond to what MotionTypes so we know what icon and color to show.

For example: "drove_alone" -> "CAR" "shared_ride" -> "CAR" "bus" -> "BUS" "walk" -> "WALKING" "air" -> "AIR_OR_HSR"

---

The "label options" for MODE just don't give us enough information:

...
  {"key":"walk", "met_equivalent": "WALKING", "co2PerMeter": 0},
  {"key":"bike", "met_equivalent": "BICYCLING", "co2PerMeter": 0},
  {"key":"e-bike", "met": {"ALL": {"range": [0, -1], "mets": 4.9}}, "co2PerMeter": 0.00728},
  {"key":"scootershare", "met_equivalent": "IN_VEHICLE", "co2PerMeter": 0.00894},
  {"key":"drove_alone", "met_equivalent": "IN_VEHICLE", "co2PerMeter": 0.22031},
  {"key":"shared_ride", "met_equivalent": "IN_VEHICLE", "co2PerMeter": 0.11015},
...

So it would be nice to have these include a mapping to our MotionTypes -- but we also want the mode choices to be configurable. What if someone wants to include some mode we've never done, like "go_cart", which doesn't align with any of our MotionTypes? Should they be able to control how it will appear by defining what color and icon it will display as?

[shankari]

I see. So once the trip is labeled, the mode indicator above the map can reflect the labeled mode. And we will display the trip on the map as unimodal from that point onward.

I am not sure we need to replace the mode indicator above the map. Can't we just make the mode indicator disappear if the user has selected the label?

So the mode indicator just represents the sensed mode.

• If the user hasn't labeled, it is what we will use.
• If the user has labeled, we ignore the sensed mode

That will also allow us to use a standard color for all "user-specific" labels without worrying about what the label is. Making new custom colors and icons for user modes is a real problem because that users can also specify their own modes (using the other option)

[JGreenlee]

Can't we just make the mode indicator disappear if the user has selected the label?

That will also allow us to use a standard color for all "user-specific" labels without worrying about what the label is.

While that would work, it doesn't seem like what users would expect. I think we would get questions of people asking where the map icons went and why all their labeled trips are the same color. People would still want to see different colors for the modes they labeled.

The colors corresponding to different modes is nice because I can scroll through my trips and see how many of them are blue (walking) and how many are driving (red), and I can visually gauge the distribution of what modes I am using. If they all become the same color after being labeled, the travel diary becomes significantly less useful (at least in the ways I use it in my daily life).

[JGreenlee]

If the inferred mode was "BIKE", showing green, and I confirmed it, wouldn't it be weird for the map to change to a different color and the bicycle icon to go away?

It would feel like "The guess was right, and I just said it was right. So why did everything just change?"

[Abby-Wheelis]

I do like the different colors, especially for multimodal trips. It sounds pretty complex to keep up with in terms of accommodating all combinations of "matched/did not match" inferred, so I'd understand if we had to scale it back temporarily. Maybe we could remove the icons upon labeling but not change the map colors? Or I like the idea of a toggle on the detail screen to keep the inferred labels accessible.

But as an example - I took a trip yesterday where I walked home, then got in my car to drive somewhere else. Seeing that 11% was walk and 89% was drive is useful, and it's nice to see where the switch happened (my home or I'd attach a screenshot).

I also really like the idea of (someday) having some explanation of our process exposed to the user. I'm just generally pro-explaining AI processes, and if we have the opportunity to have a non "black box" experience through out app that would be awesome!

[shankari]

ok, here's an alternate proposal:

• we map pre-defined label options to base modes and (by implication) colors
• these are stored in the config, just like the CO2 factor and the MET
• if somebody wants to create custom labels for their program/study, they have to tell us the base mode as well
• for user-defined labels, we will not know what the base mode is so we effectively map it to a base mode of UNKNOWN (can have it be USER_LABEL_UNKNOWN with a slightly different icon and color)

with this option:

• we can handle colors
• we can handle the mode shown in the map by either showing the base mode or leaving it out entirely

I also really like the idea of (someday) having some explanation of our process exposed to the user. I'm just generally pro-explaining AI processes, and if we have the opportunity to have a non "black box" experience through out app that would be awesome!

I think that would be a great research topic for one or more of you to explore (TRB paper for next year?!)

One advantage that we have is that, because we are open source, we don't have to keep "the algorithm" secret. It is already out there for anybody with the technical capability to see.

I think that the real challenges are:

• making it accessible to people without technical training
• reducing the complexity of the models. even for technical people, you use a random forest - it basically generates many hundreds of rules. Hannah generated the tree visualization for one users' data when she worked on label assist (I think it was mine) and we couldn't grok it fully. How do we actually explain that to users on a phone screen?

[Abby-Wheelis]

I heard back from a user that had expressed confusion at the new screen, and I think the changes in your PR will address their feedback. When I asked them about our proposal, they said:

• they were confused when they first saw the changes
• the confusing trip was a bike ride, detected as car, so they guessed they must have been going fast
• they think changing colors would be potentially less helpful

I think the "detected" string you've added next to the icon is great, and will help with users just being confused as to where that information comes from. And the colors do change, which it sounds like this user might not like, but the change from detected -> mode they inputted seems really intuitive to me especially since it's basically instantaneous to the user labeling a mode.

[JGreenlee]

This was implemented in https://github.com/e-mission/e-mission-phone/pull/1023 and has been on master for a while