Introduce Inferred Labels for Stacked Bar Charts

[iantei]

A. Introduce processing functionalities in scaffolding.py for inferred labels.

1. Introduce load_viz_notebook_inferred_data(), filter_inferred_trip() and expand_inferredlabels() for processsing, filtering and expanding inferred labels.
2. map_trip_data() to extract the mapping functionality.

[iantei]

Test Scenario:

Program: cortezebikes

Both generic_metrics and mode_specific_metrics notebook ran successfully, and the results look good.

Notebook execution for generic_metrics and mode_specific_metrics:

``` (emission) root@c5aa29285331:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb (emission) root@c5aa29285331:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default /usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="? if r.status_code is not 200: About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/ label_options is unavailable for the dynamic_config in cortezebikes Running at 2024-08-19T23:52:08.740014+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (, ) Running at 2024-08-19T23:52:08.783100+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:52:21.884262+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:52:30.599255+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:52:39.148847+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:52:47.216309+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:52:55.290742+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:03.383849+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:11.496653+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:19.128968+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:26.599253+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:34.157006+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:41.196433+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:48.375874+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:53:55.613434+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:54:02.772060+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] Running at 2024-08-19T23:54:09.801942+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')] (emission) root@c5aa29285331:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_metrics.ipynb default /usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="? if r.status_code is not 200: About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/ label_options is unavailable for the dynamic_config in cortezebikes Running at 2024-08-19T23:55:37.597376+00:00 with args Namespace(plot_notebook='mode_specific_metrics.ipynb', program='default', date=None) for range (, ) Running at 2024-08-19T23:55:37.638438+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:55:45.093898+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:55:50.707597+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:55:56.760308+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:02.884374+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:09.161714+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:14.766870+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:19.879560+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:24.874631+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:30.514902+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:36.168544+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:41.120830+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:46.193490+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:51.192843+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:56:56.136979+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] Running at 2024-08-19T23:57:01.152459+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)] (emission) root@c5aa29285331:/usr/src/app/saved-notebooks# ```

Results:	Charts Type	Charts
All Stacked Bar Charts
Number of Trips with Table

[Abby-Wheelis]

The charts look great! I'll look at the code next, but I do have one quick question - could a given trip be in the sensed, inferred, and labeled charts? Some of these look like labeled & inferred add to about 100%, but that is probably coincidence

[iantei]

The charts look great! I'll look at the code next, but I do have one quick question - could a given trip be in the sensed, inferred, and labeled charts? Some of these look like labeled & inferred add to about 100%, but that is probably coincidence

We can label a trip detected as certain mode of commute to be the same or different as the one detected. I am not sure about how labeled & inferred add up to 100.

[shankari]

I can take a look at the code, but some high level comments just by looking at the charts first:

• we should have the inferred mode in between the labeled and sensed. In general, we expect that the number of trips considered will be labeled < inferred < sensed, with sensed at 100%. We display the bars in that order for clarity
• I really question the numbers generated for the inferred modes. Per the charts above, labeled e-bike trips were around 33%, and sensed bicycling trips were around 5%. But 75% of inferred trips were apparently e-bike trips. That seems very unusual and I would like to see a deeper explanation for it before reviewing further.

I also took a brief look at the code, and I think that the current inferred_labels is only inferred labels. So if a trip had both inferred and user labels, we would use only the inferred labels even if they were wrong. That seems wrong and is certainly going to be confusing to users. The inferred labels bar should actually be "labeled and inferred" and we should only use the inferred labels for trips where they exist, but there is no user label