May Dashboard Cleanup

[Abby-Wheelis]

Beginning this PR to address bugs and cleanup in the public dashboard, see list starting here: #1051

With respect to the error handling with the survey responses, I believe this is fine, it's been working in the notebooks (and on production, noted since dfc-fermata has no data at the moment), but I will keep an eye on it. I am seeing the behavior that only one chart appears in the notebook, but all of the calls to generate_missing are going through, and I can see the figures saved in plots.

@shankari re:

for likert questions, use the related text ("neutral" or "strongly agree") instead of "3" or "5"

We currently have: "1, 2, 3, 4, 5", if we used the related text, it would be "disagree, -, neutral, -, agree" due to the formatting of the surveys to appear nicely in the app. I could check for the dashes and give us "disagree, 2, neutral, 4, agree" - is that any better? I think it could work if we make sure it is order that way in the legend at least - haven't planned how that would work yet but I'm sure it's possible.

[Abby-Wheelis]

I think it could work if we make sure it is order that way in the legend at least - haven't planned how that would work yet but I'm sure it's possible.

Trying this, and I am able to order them in the dataframe, but when I pass it into the plot_and_text_stacked_bar_chart function, they get ordered again... can I take this ordering out?

-- commented out for now, how does this look? If there is just a subset of these labels, that is what is shown as well

[shankari]

@Abby-Wheelis I would suggest the following:

• include the numeric value for all the labels to make it clear (e.g. Disagree (1), and Neutral(3))
• You can pass in a command line argument on whether to sort or not. If you feel comfortable, I would actually suggest moving the sorting to the pre-processing - e.g. df.groupby('...').agg([...]).apply(calc_pct).sort_values(...). The other metrics would then include the sorting step, and these would not.

[Abby-Wheelis]

include the numeric value for all the labels to make it clear (e.g. Disagree (1), and Neutral(3))

Done!

You can pass in a command line argument on whether to sort or not. If you feel comfortable, I would actually suggest moving the sorting to the pre-processing

I can absolutely do this, but for my own understanding, why do we need to order the metrics in the other charts? The reason that comes to front of mind would be so that they are in the same order between charts?

[shankari]

I can absolutely do this, but for my own understanding, why do we need to order the metrics in the other charts? The reason that comes to front of mind would be so that they are in the same order between charts?

I think it is just to draw the eye towards the biggest bar; and have the ordering be in a logical order instead of arbitrary. You can try removing it for the other metrics and see what they look like.

Having said all that, I think that we may want to change that sorting anyway when we unify the colors and have them be mapped to the base modes. I will anticipate that we will not want to sort at that point since we will want to have the same ordering for all the charts, which would ideally be based on similar base modes. That's why I am ok with the short-term solution of just passing in an additional argument to the plot function.

[Abby-Wheelis]

Piloting a few of the options we've been talking about:

Ordered:

Unordered:

By count with high participation:

By count with low participation:

[Abby-Wheelis]

side-by-side html tables with a small css change, could increase the padding if wanted, but not by much to avoid cuttoff

we cold also make sure that scroll left-right is available, which will be important when we add inferred (3rd table)

[Abby-Wheelis]

What it would look like if all of the survey charts were together:

Long survey:

I think this would look fine if we split the survey question names into two or three lines ... worried about the longer keys since it's more difficult to put them below now.

[shankari]

For the TripConfirmSurvey, you should also have the question on the side of the bar. Right now, you have one question at the top (which is really only the question for the last bar) and "Responses" on the side of all the bars.

I do think that the DfcEvReturnTrip looks much nicer this way. It gives a sense of the set of questions asked for this survey. And it looks like both the ReturnTrip and the RoamingTrip have the question "You are satisfied with your choice to take an EV on this trip" (Return shows up here, Roaming shows up on the public dashboard), so grouping the questions by the survey would help clarify that as well.

So assuming you can split the questions into multiple lines, I would suggest that we do group by survey. We should also make sure that the questions for the "Insufficent data" bars are also shown.

[Abby-Wheelis]

For the TripConfirmSurvey, you should also have the question on the side of the bar. Right now, you have one question at the top (which is really only the question for the last bar) and "Responses" on the side of all the bars.

Will do! Sorry, this was one of the first drafts then I kept making changes before I took the 2nd screenshot, I should have clarified that in the comment!

line split the questions:

question shown even if no data:

Remaining tasks:

• [x] debug the custom labels that are broken on deployment - bad data?
• [x] add the new labeling rate visualization
• [ ] test on multiple datasets to look for bugs and:
  • [ ] make sure the labels of "users" vs "tester and participants" are consistent across
  • [ ] make sure the legends work and look nice on all the stacked bar charts
  • [ ] test with sensed but no labeled data to make sure I patched everywhere

[Abby-Wheelis]

The issue with caebike was the missing Other label -- laos has Other as a mode, caebike has Other mode which is why only some were broken, I have added the mode manually as a part of the label generation process

[Abby-Wheelis]

Adding a new plot to show the difference between labeled and sensed over time - caebike cortezebikes

[Abby-Wheelis]

Testing Done:

ca-ebikes (custom labels)

    

dfc-fermata (low data, pushes limits of survey space)

 ^ I wish these could be dynamically sized, and need to figure out why the axis label is pushed so far below the axis itself

washingtoncommons (more "normal" survey data, also to test command line since dfc has show tests set to false now)


TODO:

• [x] test with staging or another high-variety dataset to make sure the legends are working
• [ ] clean up the label on the x axis for survey charts (very big lower offset)
• [x] run all of the notebooks once with sensed data but no labeled data to check my patch
• [x] see if I can get the new labeled vs total chart to show for the surveys - it currently relies on the mode labels since I put it in with the rest of generic_timeseries but could probably adapt fairly easily and I think dfc would really like to see this

@shankari Hoping to have this fully ready for review by tonight, getting down to the last details!

[Abby-Wheelis]

I think this works with the legend always by the side - makes sure the mapping is easily visible with each of the bars

The way we adjust the positioning of the x axis label is with: fig.supxlabel('Proportion (Count)', fontsize=20, x=0.5, y= ax.xaxis.get_label().get_position()[0] - .62, va='top')

If I lower this number, I can get it to fit right with all of the bars in the surveys - but for just 1 or 2 bars it overlaps the graph. Is there a better way to position it?

Added the labeled vs sensed time series to the survey deployments!

[Abby-Wheelis]

Last few changes fixing the html side-by-side and an extra python version change:

This is ready for a review!

[shankari]

fig.supxlabel('Proportion (Count)', fontsize=20, x=0.5, y= ax.xaxis.get_label().get_position()[0] - .62, va='top')

@Abby-Wheelis In general, I am not a big fan of tweaking X and Y positions because, as you found out, it can be complicated to deal with overlaps. Is there a reason that we have to do this instead of just setting the x label? Since we use sharex and sharey, only one x label will be displayed. We don't need a special supxlabel call.

[Abby-Wheelis]

Is there a reason that we have to do this instead of just setting the x label? Since we use sharex and sharey, only one x label will be displayed. We don't need a special supxlabel call.

I was not aware of a specific reason - a simple set_xlabel works much better!