Closed iantei closed 6 months ago
iantei
commented
8 months ago iantei
commented
8 months ago Overall Summary of above commits [so far]:
plots.py that creates html file as alt text showing table of mode, count, proportion and trip type in a table.get_quality_text_sensed function. iantei
commented
8 months ago Test Scenario:
Dataset used: openpath_prod_cortezebikes and openpath_prod_usaid_laos_ev
Execution Steps:
Changes in the `docker-compose.yml`
```
services:
notebook-server:
environment:
- DB_HOST=mongodb://db/openpath_prod_cortezebikes
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- STUDY_CONFIG=cortezebikes
```
Execution of Notebooks:
Generic Metrics:
```
(emission) root@0a163aa8efa1:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@0a163aa8efa1:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-03-01T22:41:21.026162+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Changes in the `docker-compose.yml`
```
services:
notebook-server:
environment:
- DB_HOST=mongodb://db/openpath_prod_usaid_laos_ev
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- STUDY_CONFIG=usaid-laos-ev
```
Generic Metrics
```
(emission) root@3c9e234e371d:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@3c9e234e371d:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-03-01T23:08:03.417412+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Chart Type | Chart |
|--------|--------|
| Default loading | |
| Five Stacked Bar Charts |
|
| With Tables |
|
| Number of Trips |
|
| Number of Trips Under 80th |
|
| Commute Based |
|
| Purpose Based |
|
| Distance Based |
|
| Chart Type | Chart |
|--------|--------|
| Default Loading | |
| Five Stacked Bar Charts |
|
| With Tables |
|
| Number of Trips |
|
| Number of Trips Under 80th |
|
| Commute Based |
|
| Purpose Based |
|
| Distance Based |
|
All the metrics from the dropdown menu shows different charts properly.
iantei
commented
8 months ago Comparing the data between the previous and current version of changes:
| Chart Type | Stacked Bar Chart | Pie Chart | Remarks |
|---|---|---|---|
| Number of Trips | The labels have same value and proportions except Other. | ||
| Number of Trips (Under 80) | The labels have same value and proportions except Other. |
Note: Other is expanded in case of Stacked Bar Charts.
iantei
commented
8 months ago Comparison of other charts apart from ones represented in Stacked Bar Charts:
Execution Details:
```
(emission) root@9d8152abc6f3:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@9d8152abc6f3:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-03-02T03:33:59.453960+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
```
(emission) root@31aaea26bb01:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@31aaea26bb01:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-03-02T03:19:47.078876+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Old (from current main branch) | New (with current PR changes) |
|---|---|
All the charts in both the cases are identical.
iantei
commented
8 months ago Comparison after above changes:
For Laos-ev:
| Before | After |
|---|---|
| Dropdown menu for Laos-ev: | Menu for Laos-ev |
|---|---|
Notes on updated changes:
"Number of trips by purpose" Kilometers for each mode to Total trip length (kilometers) covered by each mode.Total Trip Length ( in kilometers ) covered by mode to Total trip length (kilometers) covered by mode in dropdown.ax.set_ylim() to constrain the size of the bar. process_data_frame() function.iantei
commented
7 months ago With the latest changes incorporated to shorten the Stacked Bar Charts' labels.
| Program/Study | Chart |
|---|---|
| Laos-ev | |
| Cortezebikes |
Abby-Wheelis
commented
7 months ago Once you clear up the label on the purpose chart for e-bike trips I think this is ready to move on in review!
iantei
commented
7 months ago Execution of generate_plots scripts:
```
(emission) root@5471faec1c22:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@5471faec1c22:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-03-22T15:44:21.516485+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
```
(emission) root@2f3681b20a1d:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@2f3681b20a1d:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-03-22T15:55:48.951858+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Program/Study Type | Chart Description | Charts |
|---|---|---|
| Cortez | Default | |
| Cortez | All Stacked Bar Charts | |
| Cortez | Metric Dropdown | |
| Laos | Default | |
| Laos | All Stacked Bar Charts | |
| Laos | Metric Dropdown |
Changes update:
study_type i.e. do not generate specific charts for study, which are only related with program.
shankari
commented
7 months ago @iantei I think we should keep the mode specific metrics separate as a separate notebook.
Given that this is going to be separated out again as part of the inferred mode changes, let's split it out here (which should also make this PR cleaner) and not have a lot of commit churn
iantei
commented
6 months ago Test Scenario:
Dataset used: openpath_prod_cortezebikes
Execution of generate_plots.py for the following notebooks:
generic_metrics.ipynbgeneric_timeseries.ipynbmode_specific_metrics.ipynbmode_specific_timeseries.ipynbenergy_calculations.ipynb
1. **generic_metrics** notebook
```
(emission) root@6ad364f2ec58:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@6ad364f2ec58:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-04-16T13:24:39.946623+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Results:
| Chart Type | Chart |
|--------|--------|
| Default Charts | |
| All 100% Stacked Bar Charts |
|
| 100% Stacked Bar Chart with Additional Text Info |
|
| All Metrics |
|
| Number of Trips For specific date |
|
iantei
commented
6 months ago Re-execution of generate_plots.py scripts for cortezebikes to validate changes after ed835a5:
1. `generic_metrics.ipynb` notebook:
```
(emission) root@06e955fd4892:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@06e955fd4892:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-04-16T18:13:05.017519+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Results:
| 100% Stacked Charts |
|---|
iantei
commented
6 months ago I also noticed that "other" and "not a trip" are the same color in some of the charts, we should find a way around that
There was some existing overlapping of color palette with "Other" and "No Travel". Upon investigating, they had same color palette. Moreover, we also had unnecessary addition of "Other" since purpose_labels already has it. Now, we have combined colors_mode, colors_sensed and colors_purpose into a single one, we don't need additional Other, which in turn resolved the above issue.
Change in the visualization of 100% Stacked Bar Charts:
| 100% Stacked Bar Charts |
|---|
Abby-Wheelis
commented
6 months ago Looks good to me!
shankari
commented
6 months ago I am willing to hear the case for why passing in an array of dataframes into a single, complex function is better than the pseudocode above, but you have to make the case
iantei
commented
6 months ago I would like to present my understanding and reasoning for choosing a single function with array of data frames.
A. How’s this different from the previous implementation?
B. Reasoning for the current design:
dfs into a list and pass it to a single function because I wanted to avoid hardcoded values for ax[0] , ax[1], and nrows value, which has higher chances of error while making changes in the code.Regarding addition of inference based changes, apart from changes to extract inference data, from the Stacked Bar Chart perspective - it would simply mean adding a block of code:
new_df with an existing function process_data_framenew_df to all_data_frames (used to display chart) and all_data_frames_expanded (used to display alt_html table).bar_counts means we don't need to be mindful about changing the nrows values manually as we increase or decrease the number of horizontal bars in fig, ax = plt.subplots(ncols=1, nrows=2). Also, the value of nrows would be dependent on the df. For example, if the df for inferred label is empty (which cannot be anticipated), we would revert back to only showing sensed and labeled bars. In this case, it helps for nrows to be set dynamically than with fixed value?E.g.
if not expanded_it.empty:
df_inferred_tc_expanded, df_inferred_tc = process_data_frame(expanded_it, 'Mode_confirm', “Inferred from historyr\n")
if not df_inferred_tc.empty:
bar_count += 1
all_data_frames.append(df_inferred_tc)
all_data_frames_expanded.append(df_inferred_tc_expanded)
else:
print("df_inferred_tc is empty.")
else:
print("expanded_it is empty.")C. We have different all_data_frames and all_data_frames_expanded because we are displaying merged Other labels, while in alt_text and alt_html we are presenting the overall chart.
D. The handling of expanded_ct, expanded_it and expanded_sensed_ct seems a bit overwhelming change on the notebook, but this helps to this displays the bar only for which the data frame is available.
This was added specifically to account when data_eb (mode specific df) would be empty, while expanded_ct was not empty. This check might be useful when we add inferred labels.
E. Question about the proposed design:
In the main branch, currently we have different functions pie_chart_sensed, pie_chart_mode and pie_chart_purpose which are identical.
Abby-Wheelis
commented
6 months ago Can we use the length of the list as the number of bars? Would reduce an argument
Would refactoring such that the list of dfs is passed into a "make stacked bar chart" function, and then we call a "make bar" function for each df in the list? Would that make it any more clear while still not needing to hardcode which axis position a given bar will take? This might create a little bit of overhead in creating a new function, but would it be worth it to make the stacked bar chart creating more digestible?
iantei
commented
6 months ago Notes for changes from the meeting:
ax.text when the data frame is missing, and the bar is not generated, instead of no bar(s).iantei
commented
6 months ago Test Scenario:
Dataset used: openpath_prod_cortezebikes
1. Generic metrics notebook
```
(emission) root@32f4a8b63cc2:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@32f4a8b63cc2:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-04-24T20:14:20.790286+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Result: | 100% Stacked Bar Charts |
|---|---|
iantei
commented
6 months ago Assuming, we the data frame is unavailable, this is the way the unavailable bar is represented.
| Unavailable Data frame |
|---|
iantei
commented
6 months ago Test Scenario:
Dataset used: openpath_prod_cortezebikes
Generic notebook:
```
(emission) root@8eb38e8e7756:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@8eb38e8e7756:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-04-25T12:13:40.017400+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Result: | 100% Stacked Bar Charts - Cortezebikes |
|---|---|
Note: No changes from the above. Testing with refactoring and merging with main.
iantei
commented
6 months ago Test Scenario:
Dataset used: openpath_prod_cortezebikes
1. Generic Metrics notebook
```
(emission) root@28d5c8478cbf:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@28d5c8478cbf:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-04-26T20:04:46.756894+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Results:
| Chart Type | Chart |
|---|---|
| 100% Stacked Bar Chart | |
| 100% Stacked Bar Chart without data | |
| Sample alt_html | |
| Sample alt_text |
shankari
commented
6 months ago @iantei I hope that the code examples clarify what I had in mind. We have been working on this for months now, and we do have a deadline coming up. Let's go through one more review cycle, and if it is not done, I will take over and finish up. Even if I do take over and finish up, I would appreciate your testing the code since my time is limited.
shankari
commented
6 months ago @iantei note that this still has a merge conflict that you need to resolve
shankari
commented
6 months ago @iantei please resolve the merge conflict. After that, I will take over since it looks like it is hard for you to implement the outline in https://github.com/e-mission/em-public-dashboard/pull/123#discussion_r1583383009
shankari
commented
6 months ago @iantei just confirming that you are done with your changes and this is now back with me....
shankari
commented
6 months ago Finished the first heavy lift. We now have a nice-ish structure in which we plot and generate the text and HTML files for each bar at a time, and then pull everything together to generate the full figure/full text files. We also now handle code errors differently from insufficient data - insufficient data is at the bar level, while code errors are at the cell level.
Testing done with errors (prior to loading data).
| No data | code error (after introducing an intentional code error) |
|---|---|
ntrips_total_default_no_trips.txt ntrips_total_default_no_trips.html.gz
ntrips_total_default_code_error.html.gz ntrips_total_default_code_error.txt
shankari
commented
6 months ago @iantei, please see https://github.com/e-mission/em-public-dashboard/pull/123/commits/12b00e3b48c7572d7f9309ecc834bfc9d1b266a7 for what I had in mind
I am now going to do some more cleanup, finish converting everything in generic_metrics and test with real data.
I would like you to convert mode_specific_metrics and finish testing.
shankari
commented
6 months ago Tested with real data, and (with some changes), I get a valid plot
However, note that:
Comparing with prior results in https://github.com/e-mission/em-public-dashboard/pull/123#issuecomment-2080079104
shankari
commented
6 months ago @iantei I am done with my changes. I would like you to make the corresponding changes to the mode specific metrics and then to test before pushing the code. Please let me know ASAP if you are not able to do this so that I can plan on doing so and unblocking @Abby-Wheelis
I have some higher-level thoughts about the structure which I will put into the issue.
@Abby-Wheelis this should be fairly solid wrt the underlying structure, with the caveat that I might want to move out the filtering into the cell (see my upcoming note in the issue). Please digest my notes about the structure and LMK if you have any questions before you finalize the survey work
iantei
commented
6 months ago @shankari I have my Final exam tomorrow, and also tomorrow is my last day to access the NREL laptop prior to re-joining later. I will spend some time today, to pull the latest changes, and make changes required for mode_specific_mode notebook and test it. I will let you know about the progress and update over the end of the day.
shankari
commented
6 months ago @iantei if you are not able to do this given school pressures, I understand. I would just like to make sure that I know ahead of time so that I can pick up the task and unblock @Abby-Wheelis
Abby-Wheelis
commented
6 months ago @iantei @shankari I don't think I'll be able to get to this until Friday night at the very earliest, I am trying to rebalance my time btwn OpenPATH and other tasks for the rest of the work-week, but will make sure I read through the updates and ask questions if I have them before the end of the workday tomorrow! -- just to give an idea of where I am and hopefully help with planning of timing.
shankari
commented
6 months ago @Abby-Wheelis I will be working on the weekend; if we coordinate our schedules, you don't have to get the questions done by tomorrow. We can just sync up on the weekend.
iantei
commented
6 months ago Test Scenario:
Dataset used: cortez-ebikes
```
(emission) root@f75d26c8b23b:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@f75d26c8b23b:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-05-02T21:04:31.496234+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| Results: | 100% Stacked Bar Charts |
|---|---|
Remarks: generic_metrics, mode_specific_metrics and generic_metrics_sensed notebooks are all executing generate_plots without any issues.
iantei
commented
6 months ago @shankari Encountered two issues:
Observed the issue with alt_html which does not show the table of information, whenever there is a single bar in Stacked Bar Chart. These issues are observed with Purpose, Commute related charts, and same is observed with e-bike related charts since they have single bars too.
| Single Bar Stacked Bar Chart HTML error |
|---|
I don't have a bandwidth to work on diagnosing the issues today. I might be able to look into it after my exam tomorrow.
shankari
commented
6 months ago Observed the issue with alt_html which does not show the table of information, whenever there is a single bar in Stacked Bar Chart. These issues are observed with Purpose, Commute related charts, and same is observed with e-bike related charts since they have single bars too.
It is not that there is no table, there is a single letter ("S"). This is because I hardcoded the alt-text generation assuming that there was a 2x2 array. So I use text_results[0][0] for the alt-text and text_results[0][1] for alt-HTML for the first plot.
But if there is only one set of results, then text_results[0][0] is the first character of the string instead of the string 😄
concat_alt_text = plot_title + text_results[0][0] + text_results[1][0]
...
<!DOCTYPE html>
<html>
<body>
<p>{plot_title}</p>
<p>{text_results[0][1]}</p>
<p>{text_results[1][0]}</p>
</body>
</html>Will fix to take into account the number of axes
shankari
commented
6 months ago @iantei fixed
iantei
commented
6 months ago @shankari The alt_html was not showcasing the information in table structure. I assume we are still planning to showcase it in tabular format. Here's the current sample for the alt_html with multiple and single bar - for Number of Trips and Commute Trips below: |
alt_html table |
|---|---|
Note: I just noticed the top row header is missing in table.
iantei
commented
6 months ago Test Scenario:
Dataset: cortezebikes
```
ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-notebook-server-1 /bin/bash
root@9f119352305c:/usr/src/app# source setup/activate.sh && cd saved-notebooks
(emission) root@9f119352305c:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@9f119352305c:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-05-04T00:39:51.554550+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Result:
alt_html validation for single and multiple bars:
| With alt_html |
|---|
shankari
commented
6 months ago @iantei it looks like the real error was returning alt_html instead of html_content, and it is now fixed. Do you see any other errors?
iantei
commented
6 months ago I'm testing with usaid-laos-ev dataset. I didn't observe any errors apart the one mentioned above - which has been fixed.
iantei
commented
6 months ago Test Scenario:
Dataset: usaid-laos-ev:
Generic Metrics notebook:
```
(emission) root@3c25c8c8a6ab:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@3c25c8c8a6ab:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-05-04T03:32:00.426215+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Note: mode_specific_metrics notebook's generate_plots script was not executed since usaid-laos-ev is a study.
Results:
Consolidated Result of Stacked Bar Chart with alt_html: |
Chart |
|---|---|
Note: There is an overlapping choice of color palette between Motorcycle and AIR_OR_HSR, which can distinction through use of different color legends.
iantei
commented
6 months ago Comparing Laos-Chart comment to the above charts. Few observations:
Labeled by user Trip Types are identical except the Other label's value which has been condensed in above charts.Overall, looks good.
iantei
commented
6 months ago @shankari Encountered few issues: Different variation of error description table/text loading for charts when the Month is selected as: 5/2024
Dataset used: openpath-prod-usaid-laos-ev-snapshot-dec-20
| Chart Name | Chart |
|---|---|
| Number of Trips by Purpose | |
| Number of Commute Trips | |
| Under 80th Percentile | |
| Total Trip Length |
iantei
commented
6 months ago UPDATE: @shankari I have lost access to the NREL system account.
shankari
commented
6 months ago @iantei That is largely expected because there is no data for 5/2024. As you can see, there were no trips in that month. We should not be displaying the backtrace instead of the missing data (see upcoming commit) but it is not really a showstopper since it wouldn't have worked anyway.
A bigger showstopper is the one that I highlighted here: https://github.com/e-mission/em-public-dashboard/pull/123#issuecomment-2089760981
While I am fine with working on code structure, I was really really hoping to not have to tweak pandas horizontal bar generation and spacing. My hope was that the visualization intern(s) would make the charts look pretty and I only needed to make the code look pretty.
Abby-Wheelis
commented
6 months ago I'm starting to catch up with these changes to hopefully be able to incorporate the surveys smoothly, and on this part of the showstopper mentioned above -
The bar extends beyond 100% with a gray color that is not in the legend
I think this is the plot itself and not part of the bar (this is the background for the chart area and you can see a faint white gridline) so if we want it removed for aesthetic reasons we can do that but I don't think it represents a value/needs to be in the legend
shankari
commented
6 months ago I think this is the plot itself and not part of the bar (this is the background for the chart area and you can see a faint white gridline) so if we want it removed for aesthetic reasons we can do that but I don't think it represents a value/needs to be in the legend
I agree that it should not be in the legend. But I am not sure that it so easy to fix because we are computing the blocks to generate the horizontal bar. Is there a reason why we didn't just use pandas DataFrame.bar with stacked=True?
Also, the second point in the showstopper (the legend for the labeled trips is too big and overlaps the sensed) will be even more true for the survey changes, correct? Do you have any thoughts on how to change that? Maybe a horizontal legend instead of vertical?
Abby-Wheelis
commented
6 months ago Also, the second point in the showstopper (the legend for the labeled trips is too big and overlaps the sensed) will be even more true for the survey changes, correct? Do you have any thoughts on how to change that? Maybe a horizontal legend instead of vertical?
I think that we'll put it horizontally, below the chart itself since the labels may have much longer text also, I anticipate this being the most difficult to get worked out visually when it comes to the surveys.
As far as using pandas Dataframe.bar I'm not sure if there is any particular reason why we did it this way, I have a general sense that the matplotlib method is a little easier to customize, but I haven't tried the pandas method, maybe that would work.
shankari
commented
6 months ago After improving error handing, testing done: set the date to one for which we have no data, including for the ones which showed a backtrace in https://github.com/e-mission/em-public-dashboard/pull/123#issuecomment-2094004598
Both alt_text and alt_html files have been created for the newly created stacked bar charts, not for the older bar charts
$ ls -al plots/*_2020_11* | grep "May *4"
-rw-r--r-- 1 kshankar staff 85045 May 4 18:59 plots/average_miles_mode_confirm_2020_11_default.png
-rw-r--r-- 1 kshankar staff 342 May 4 18:59 plots/average_miles_mode_confirm_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 355 May 4 18:59 plots/ntrips_commute_mode_confirm_2020_11_default.html
-rw-r--r-- 1 kshankar staff 88215 May 4 18:59 plots/ntrips_commute_mode_confirm_2020_11_default.png
-rw-r--r-- 1 kshankar staff 355 May 4 18:59 plots/ntrips_commute_mode_confirm_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 81393 May 4 18:59 plots/ntrips_per_day_2020_11_default.png
-rw-r--r-- 1 kshankar staff 320 May 4 18:59 plots/ntrips_per_day_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 82616 May 4 18:59 plots/ntrips_per_weekday_2020_11_default.png
-rw-r--r-- 1 kshankar staff 324 May 4 18:59 plots/ntrips_per_weekday_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 350 May 4 18:59 plots/ntrips_purpose_2020_11_default.html
-rw-r--r-- 1 kshankar staff 86380 May 4 18:59 plots/ntrips_purpose_2020_11_default.png
-rw-r--r-- 1 kshankar staff 350 May 4 18:59 plots/ntrips_purpose_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 369 May 4 18:58 plots/ntrips_total_2020_11_default.html
-rw-r--r-- 1 kshankar staff 90649 May 4 18:58 plots/ntrips_total_2020_11_default.png
-rw-r--r-- 1 kshankar staff 369 May 4 18:58 plots/ntrips_total_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 372 May 4 18:59 plots/ntrips_under80_2020_11_default.html
-rw-r--r-- 1 kshankar staff 91797 May 4 18:59 plots/ntrips_under80_2020_11_default.png
-rw-r--r-- 1 kshankar staff 370 May 4 18:59 plots/ntrips_under80_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 339 May 4 18:59 plots/total_trip_length_2020_11_default.html
-rw-r--r-- 1 kshankar staff 84806 May 4 18:59 plots/total_trip_length_2020_11_default.png
-rw-r--r-- 1 kshankar staff 339 May 4 18:59 plots/total_trip_length_2020_11_default.txt
-rw-r--r-- 1 kshankar staff 374 May 4 18:59 plots/total_trip_length_land_2020_11_default.html
-rw-r--r-- 1 kshankar staff 93236 May 4 18:59 plots/total_trip_length_land_2020_11_default.png
-rw-r--r-- 1 kshankar staff 374 May 4 18:59 plots/total_trip_length_land_2020_11_default.txt