Closed shankari closed 9 months ago
This is likely to be particularly challenging in international contexts - e.g. https://usaid-laos-ev-openpath.nrel.gov/public/
Note, however, that Laos is not currently using custom labels - this is likely because the user truly added an custom mode (e.g. "motorcycle" or "rickshaw") by selecting "other" from the dropdown and then kept using it. We will not tackle that use case now, only custom labels specified by program admins and included in the dynamic config.
suggested steps going forward
We are creating a dictionary with key as replaced_mode: drove_alone with values as mode_clean: "Gas car, drove alone" in mapping_dictionaries.ipynb, which is stored as dic_re = dict(zip(df_re['replaced_mode'],df_re['mode_clean'])) # bin modes. This is being retrieved in generic_metrics.ipynb - %store -r dic_re.
Close, but that is for the replaced mode, not for the mode. The very first figure, for example, does not use the replaced mode at all.
The saved dic_re is being passed into scaffolding.load_viz_notebook_data(..., dic_re, ...).
Inside scaffolding.py: load_viz_notebook_data(..., dic_re, ...):
Now the passed dic_re being used in ::
expanded_ct['Mode_confirm']= expanded_ct['mode_confirm'].map(dic_re)
The above statement will map expanded_ct['mode_confirm'] to the key of dic_re. E.g. if the value of expanded_ct['mode_confirm'] is "drove alone", and the dic_re holds
so you now need to read the mapping (the equivalent of dic_re
) from the dynamic labels if the dynamic config indicates that the study/program uses dynamic labels. Submit a draft PR for that change.
The approach I'm undertaking is the following:
For the development mode, can we execute the generate_plots.py further calling the respective notebook (generic_metrics.ipynb)? While trying this it looked up for few modules from emission.storage, which it wouldn't be able to access.
So, I understand an approach would be to take up the variables directly at the starting of the notebook, and execute. Is there an alternative way other than that for a development mode testing?
There is a disparity in the mode_labels.csv and example-program-label-options.json referred from dev-emulator-program.nrel-op.json mapping. In mode_labels.csv, we have mode_confirm[drove_alone] -> mode_clean["Gas Car, drove alone"]. Similarly, in the above example-program-label-options.json file, in the translations[en] - drove_alone maps to "Gas Car Drove Alone". This mismatch would result in an issue with the plots.py: pie_chart_mode(plot_title, labels, values, file_name) function. Where for the newly mapped labels - containing keys from Mode_confirm (which have been mapped with the dynamic_labels approach than with dic_re). Calling colours[key] would result in crash - thus no Pie Chart diagram.
all_labels= ['Gas Car, drove alone', 'Bus', 'Train', 'Free Shuttle', 'Taxi/Uber/Lyft', 'Gas Car, with others', 'Bikeshare', 'Scooter share', 'E-bike', 'Walk', 'Skate board', 'Regular Bike', 'Not a Trip', 'No Travel', 'Same Mode', 'E-car, drove alone', 'E-car, with others', 'Air', 'Other'] colours = dict(zip(all_labels, plt.cm.tab20.colors[:len(all_labels)])) colors=[colours[key] for key in labels] -- this would crash, since it looks up for the key "Gas Car Drove Alone" instead of "Gas Car, drove alone".
I understand the fix would be to make appropriate changes with example-program-label-options.json to match it with mode_labels.csv.
can we execute the generate_plots.py further calling the respective notebook (generic_metrics.ipynb)? While trying this it looked up for few modules from emission.storage, which it wouldn't be able to access.
Again, this is not very useful. What did you try to do to execute the generate_plots.py
? What error did you get?
All the notebooks use modules from emission.storage
, so I don't know what you mean by this statement.
I understand the fix would be to make appropriate changes with example-program-label-options.json to match it with mode_labels.csv.
Your understanding is incorrect. Please read the high-level goals in the issue. It is not "support example-program-label-options.json
". It is "support custom labels for each program/study" - example-program-label-options.json
is just an example.
Given that the list of labels can be custom, we can no longer hardcode the list of colors. I don't see why the list of colors needs to be hardcoded in the first place given that we are mapping it to plt.cm.tab20.colors
anyway. We should just have all_labels
be the list of models in the custom list instead of a hardcoded list.
Note that the list of modes now also specifies the basemode. As a stretch goal, you can use the number of modes of each base mode to get shades of the basemode color and be consistent with the phone UI. But that is a future enhancement after we get the basic custom mode display correct.
I was validating my changes with reference to the available example for "support custom labels for each program/study" which is example-program-label-options.json
. There's a mismatch in the mapping provided in the above example file to the one present on the mode_labels.csv. Therefore, I pointed out, it'd be rightful to update the example-program-label-options.json
accordingly.
@iantei you cannot update example-program-label-options.json
. Our program partners can give us whatever they want in label-options.json
and we have to be able to support it. That is the goal of this new feature.
Please read through my comments on "hardcoded list of colors" carefully.
Testing done so far: Scenario I: walk - with default mapping Dataset used: vail_2022-05-09.tar.gz STUDY_CONFIG=stage-program
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-27T23:40:59.466627+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
```
Result:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T00:08:46.554062+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_timeseries.ipynb default
```
Result:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T00:20:59.704644+00:00 with args Namespace(plot_notebook='generic_timeseries.ipynb', program='default', date=None) for range (
Executed the following code:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_metrics.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T00:26:54.014794+00:00 with args Namespace(plot_notebook='mode_specific_metrics.ipynb', program='default', date=None) for range (
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T00:37:18.773477+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Executed the following:
```
python bin/generate_plots.py energy_calculations.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T00:46:59.894921+00:00 with args Namespace(plot_notebook='energy_calculations.ipynb', program='default', date=None) for range (
//Testing done so far: [DRAFT: Still updating] Scenario II: moped - with default mapping Dataset used: vali_2022-05-09.tar.gz STUDY_CONFIG=stage-program
Showcasing existing entries in the mongodb: ``` ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-db-1 mongo MongoDB shell version v4.4.0 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("727fcd12-c272-4a0f-9b2e-1f6a7ce489b2") } MongoDB server version: 4.4.0 --- The server generated these startup warnings when booting: 2023-09-27T23:36:15.782+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2023-09-27T23:36:16.838+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted --- --- Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc). The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product improvements and to suggest MongoDB products and deployment options to you. To enable free monitoring, run the following command: db.enableFreeMonitoring() To permanently disable this reminder, run the following command: db.disableFreeMonitoring() --- > clear uncaught exception: ReferenceError: clear is not defined : @(shell):1:1 > show dbs Stage_database 13.751GB admin 0.000GB config 0.000GB local 0.000GB > use Stage_database switched to db Stage_database ``` Results: ``` > db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"walk"}).count() 4359 ``` Executed the below script to make changes on the mongoDb: ``` var result_walk = db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"walk"}, {"_id": 1}) var resultArray = result_walk.toArray(); for (var i = 0; i < resultArray.length; i++) { var doc = resultArray[i]; var docId = doc._id; var updateResult = db.Stage_analysis_timeseries.update( { "_id": docId, "data.user_input.mode_confirm": "walk" }, { "$set": { "data.user_input.mode_confirm": "moped" } } ); if (updateResult.nModified > 0) { print(`Updated document with _id: ${docId}`); } } ``` Result of above execution: ``` > db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"walk"}).count() 0 > db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"moped"}).count() 4359 ```
Executed the following: ``` ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-dashboard-1 /bin/bash OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown ashrest2-35384s:em-public-dashboard ashrest2$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 75ac05b4ded2 em-pub-dash-dev/frontend "docker-entrypoint.s…" 2 hours ago Up 2 minutes 0.0.0.0:3274->6060/tcp em-public-dashboard-dashboard-1 61584fb3620f em-pub-dash-dev/viz-scripts "/bin/bash /usr/src/…" 2 hours ago Up 2 minutes 8080/tcp, 0.0.0.0:47962->8888/tcp em-public-dashboard-notebook-server-1 c850b9a6f83e mongo:4.4.0 "docker-entrypoint.s…" 4 weeks ago Up 2 minutes 27017/tcp em-public-dashboard-db-1 ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-notebook-server-1 /bin/bash root@61584fb3620f:/usr/src/app# source setup/activate.sh (emission) root@61584fb3620f:/usr/src/app# cd saved-notebooks/ ``` Re-evaluated the status of mongodb: ``` ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-db-1 mongo MongoDB shell version v4.4.0 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("e07e8164-0440-4afc-924c-e5dc5987f925") } MongoDB server version: 4.4.0 --- The server generated these startup warnings when booting: 2023-09-28T01:08:13.856+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2023-09-28T01:08:14.670+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted --- --- Enable MongoDB's free cloud-based monitoring service, which will then receive and display metrics about your deployment (disk utilization, CPU, operation statistics, etc). The monitoring data will be available on a MongoDB website with a unique URL accessible to you and anyone you share the URL with. MongoDB may use this information to make product improvements and to suggest MongoDB products and deployment options to you. To enable free monitoring, run the following command: db.enableFreeMonitoring() To permanently disable this reminder, run the following command: db.disableFreeMonitoring() --- > show dbs Stage_database 13.751GB admin 0.000GB config 0.000GB local 0.000GB > use Stage_database switched to db Stage_database > clear uncaught exception: ReferenceError: clear is not defined : @(shell):1:1 > db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"walk"}).count() 0 > db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"moped"}).count() 4359 ```
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T01:19:53.638888+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
I travelled for half an hour or so, now it's been updated to show the below. Refreshed the page and now it's loading properly with "Moped" data as Others
Execution:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
```
Result:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T02:09:27.333781+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_timeseries.ipynb default
```
Result:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T02:13:18.254876+00:00 with args Namespace(plot_notebook='generic_timeseries.ipynb', program='default', date=None) for range (
Executed the following command:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_metrics.ipynb defaul
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T02:22:55.108546+00:00 with args Namespace(plot_notebook='mode_specific_metrics.ipynb', program='default', date=None) for range (
Executed the following code:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
```
Result:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T02:29:17.211150+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Executed the following:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py energy_calculations.ipynb default
```
Result:
```
(emission) root@61584fb3620f:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py energy_calculations.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T02:32:10.257561+00:00 with args Namespace(plot_notebook='energy_calculations.ipynb', program='default', date=None) for range (
wrt
Executed for generic_metrics_sensed notebook: [Gives Error]
this seems to be a dup of https://github.com/e-mission/em-public-dashboard/issues/93#issuecomment-1723817828
If you can check the database and verify that you see the same NaN
pattern in the trips from the Vail dataset, I am comfortable ignoring those errors.
As you can see in the above chart, the latest pie_chart depicting "Moped" mode is not being updated. The timestamp doesn't match with the one from the log.
That is kind of bizarre. I don't see any error while running this
Dynamic labels are not available.
Running at 2023-09-28T01:19:53.638888+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2020-09-01T00:00:00+00:00]>, <Arrow [2023-09-01T00:00:00+00:00]>)
Running at 2023-09-28T01:19:54.449342+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T01:20:42.475587+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
When the notebook is run with nbclient.execute
, I believe it stores the outputs in the notebook.
generate_plots
to only run for the aggregate case (year=None, month=None
) and then open the resulting notebook to see what is going onOnce you have figured it out, of course, change generate_plots
back and re-test.
The chart not being updated is not an issue. I tried to refresh the page prior to the successful completion of the notebook script. Upon the completion of notebook script, things are working fine.
/> When the notebook is run with nbclient.execute
, I believe it stores the outputs in the notebook.
- You could try changing
generate_plots
to only run for the aggregate case (year=None, month=None
) and then open the resulting notebook to see what is going on- you could also see if the errors are for the aggregate data only or also for individual months (and then compare the notebooks if only one of them works)
Once you have figured it out, of course, change
generate_plots
back and re-test.
The notebook execution is not failing. So, I don't think it's required to re-execute the notebook. I will skip it, and proceeding with the generic_metrics_sensed issue. I got the following observation:
Steps followed:
Added a debug statement in scaffolding.py - inside load_viz_notebook_data to to look up for the NaN
pattern.
Hello: participant_ct_df.cleaned_section_summary.tail()57402 {'distance': {'ON_FOOT': 886.4937093667857}, '...
57403 {'distance': {'ON_FOOT': 610.2234223038181}, '...
57404 {'distance': {'ON_FOOT': 405.97685486691756}, ...
57405 {'distance': {'IN_VEHICLE': 4230.990793080315,...
57406 {'distance': {'IN_VEHICLE': 4255.784960891164,...
Name: cleaned_section_summary, dtype: object
This seems different.
Testing done so far: Scenario I: moped - with default mapping Dataset used: vail_2022-05-09.tar.gz STUDY_CONFIG=dev-emulator-program
Note: I am using Safari to load the localhost and see changes.
1. Made the following changes
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.dev.yml
diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml
index fdbdbb8..c2a4342 100644
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -26,7 +26,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=dev-emulator-program
ports:
# ipynb in numbers
- "47962:8888"
```
Re-launched the docker-compose.dev.yml up
Executed the following:
```
(emission) root@183bf369df9a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
```
Results:
```
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful.
Running at 2023-09-28T03:19:26.899624+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Re-executing the above steps in Mozilla to see if it's browser specific issue. Tested the above scenario, this time around by using STUDY_CONFIG=stage-program. And re-executing for the generic_metrics. Still the same issue. Upon refreshing the page, the charts are not being re-loaded. But when I go into the public-dashboard page, and launch the localhost - the new changes are reflected.
There was an already open localhost webpage, reloaded it. (left) Opened the localhost webpage from public-dashboard page. (right)
@iantei So the issue that you are highlighting is that reloading does not actually reload the metrics, but opening a new page does?
Couple of high level comments:
Last updated 2023-09-28T03:19:43 (from the chart in browser) Last updated 2023-09-28T03:56:45 (from the terminal log) Last successful execution of generic_metrics in terminal log:
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T03:56:45.000945+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2020-09-01T00:00:00+00:00]>, <Arrow [2023-09-01T00:00:00+00:00]>)
Running at 2023-09-28T03:56:45.042602+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:09.580454+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:16.480758+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:23.666826+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:30.957319+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
ii. The return type is 200, 404 for sensed ones.
I will make a note of all metrics going forward.
@iantei not just going forward. There is no evidence here that the miles graph is updated, even with the new browser tab (right image)
I would like to see a record of the miles_mode_confirm (and the other metrics) images with recent timestamps as well
Here's the command I ran along with the output.
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=..python bin/update_mappings.py mapping_dictionaries.ipynb
bash: bin/update_mappings.py: Permission denied
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T03:56:45.000945+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2020-09-01T00:00:00+00:00]>, <Arrow [2023-09-01T00:00:00+00:00]>)
Running at 2023-09-28T03:56:45.042602+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:09.580454+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:16.480758+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:23.666826+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:30.957319+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:37.393099+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:46.062188+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:57:53.451332+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:00.888240+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:07.778203+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:14.674005+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:21.531039+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:29.221692+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:36.694908+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:43.982052+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:51.249523+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:58:58.671300+00:00 with params [Parameter('year', int, value=2021), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:06.116057+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:13.464401+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:20.682934+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:27.967198+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:35.532452+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:41.757946+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:49.181585+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T03:59:56.448044+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:03.772014+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:11.061260+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:18.367641+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:25.616975+00:00 with params [Parameter('year', int, value=2022), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:33.079061+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:40.411851+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:47.713259+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:00:55.111689+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:01:02.415842+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:01:09.826848+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:01:17.238615+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:01:24.568172+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Running at 2023-09-28T04:01:31.888439+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks#
@iantei Testing of the non-sensed notebooks seems to be fine. However, I do not see the reason for the notebook with the sensed metrics to fail. Once we have established that reason, I am happy to merge.
Proceeding with the generic_metrics_sensed issue. I got the following observation:
Steps followed: Added a debug statement in scaffolding.py - inside load_viz_notebook_data to look up for the NaN pattern.
Hello: participant_ct_df.cleaned_section_summary.tail()57402 {'distance': {'ON_FOOT': 886.4937093667857}, '...
Result:
57403 {'distance': {'ON_FOOT': 610.2234223038181}, '...
57404 {'distance': {'ON_FOOT': 405.97685486691756}, ...
57405 {'distance': {'IN_VEHICLE': 4230.990793080315,...
57406 {'distance': {'IN_VEHICLE': 4255.784960891164,...
Name: cleaned_section_summary, dtype: object
This seems different.
There're three notable issues with this change right now.
Added a debug statement in scaffolding.py - inside load_viz_notebook_data to look up for the NaN pattern.
Where is the patch showing the statement you added? This does not look like python code to me
Hello: participant_ct_df.cleaned_section_summary.tail()57402 {'distance': {'ON_FOOT': 886.4937093667857}, '...
Also, it is not clear that tail()
is the right option. The NaN can occur anywhere. We are doing .apply()
on the full dataframe. You need to use something like isnan
over the dataframe.
wrt https://github.com/e-mission/em-public-dashboard/issues/89#issuecomment-1738429769
Added a debug statement in scaffolding.py - inside load_viz_notebook_data to look up for the NaN pattern.
Where is the patch showing the statement you added? This does not look like python code to me
Hello: participant_ct_df.cleaned_section_summary.tail()57402 {'distance': {'ON_FOOT': 886.4937093667857}, '...
Also, it is not clear that
tail()
is the right option. The NaN can occur anywhere. We are doing.apply()
on the full dataframe. You need to use something likeisnan
over the dataframe.
Updated the following patch code in scaffolding.py: Code changes:
def load_viz_notebook_sensor_inference_data(year, month, program, include_test_users=False, sensed_algo_prefix="cleaned"):
""" Inputs:
year/month/program = parameters from the visualization notebook
Pipeline to load and process the data before use in sensor-based visualization notebooks.
"""
tq = get_time_query(year, month)
participant_ct_df = load_all_participant_trips(program, tq, include_test_users)
print("Hello:: " + str(participant_ct_df.cleaned_section_summary.isna()))
Results:
Hello:: 0 True
1 True
2 True
3 True
4 True
...
57402 False
57403 False
57404 False
57405 False
57406 False
Name: cleaned_section_summary, Length: 57407, dtype: bool
Loaded expanded_ct with length 57407 for None
There're instances of NaN present in the dataset used. The presence of NaN could be not specifically due to its presence with vail dataset, but more with few additional manual entries done too.
0 sc_...
1 sc_...
2 sc_...
3 sc_...
4 sc_...
...
133 vail_...
134 vail_...
135 vail_...
136 nrelop_dev-emulator-study_test-nschultz-july-7
137 nrel_dev-ananta-Sep13
Name: user_email, Length: 138, dtype: object
Steps involved: 1. Launch the jupyter notebook. 2. Execute the steps manually. 3. You'll encounter a bunch of future warnings: ``` /tmp/ipykernel_1479/1013976020.py:17: FutureWarning: Not prepending group keys to the result index of transform-like apply. In the future, the group keys will be included in the index, regardless of whether the applied function returns a like-indexed object. To preserve the previous behavior, use >>> .groupby(..., group_keys=False) To adopt the future behavior and silence this warning, use >>> .groupby(..., group_keys=True) mode_counts['trip_count_smooth'] = mode_counts.groupby(['user_id','mode_confirm'])['trip_count'].apply(lambda x: x.rolling(7,1).mean()) /tmp/ipykernel_1479/1013976020.py:18: FutureWarning: Not prepending group keys to the result index of transform-like apply. In the future, the group keys will be included in the index, regardless of whether the applied function returns a like-indexed object. To preserve the previous behavior, use >>> .groupby(..., group_keys=False) To adopt the future behavior and silence this warning, use >>> .groupby(..., group_keys=True) mode_distance['distance_miles_smooth'] = mode_distance.groupby(['user_id','mode_confirm'])['distance_miles'].apply(lambda x: x.rolling(7,1).mean()) ``` But eventually all charts are generated. I have not captured all charts, but all charts are building fine.
wrt #89 (comment)
- Related with the page refresh one: seems to be resolved with Support custom label dropdowns from the dynamic config #89 (comment) and Support custom label dropdowns from the dynamic config #89 (comment). The metrics are being recomputed properly, we can't fix browser caching.
- Related with generic_metric_sensed: this still needs investigation
- Related with mode_specific_timeseries: I had missed this! It also still needs investigation.
There are instance of NaN observed in the dataset used, therefore this is similar to the one we've earlier encountered. The issue (2) is now addressed.
I am able to execute it on notebook manually. (For aggregated data i.e. year, month=None, )
@iantei This is an interesting data point, and may help you debug the automated run. But you need to get the automated run to work successfully before we can merge.
Note that the failure is not for the aggregated data, but for the first month-specific run
Running at 2023-09-28T02:29:55.157467+00:00 with params [Parameter('year', int, value=2020), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={})]
Maybe you should try to test that?
Yes, it's failing when I selected Year = 2023, Month = 9.
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Cell In[9], line 1
----> 1 quality_text = scaffolding.get_quality_text(expanded_ct, mode_counts_interest, mode_of_interest, include_test_users)
NameError: name 'mode_counts_interest' is not defined
I will investigate into this tomorrow.
Dataset used: ``` 0 sc_... 1 sc_... 2 sc_... 3 sc_... 4 sc_... ... 133 vail_... 134 vail_... 135 vail_... 136 nrelop_dev-emulator-study_test-nschultz-july-7 137 nrel_dev-ananta-Sep13 Name: user_email, Length: 138, dtype: object ``` I added the following patch code for debugging: ``` def load_all_participant_trips(program, tq, load_test_users): participant_list = get_participant_uuids(program, load_test_users) all_ct = load_all_confirmed_trips(tq) print("Hello: length of all_ct is " + str(len(all_ct))) # CASE 1 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if len(all_ct) == 0: return all_ct participant_ct_df = all_ct[all_ct.user_id.isin(participant_list)] print("After filtering, found %s participant trips " % len(participant_ct_df)) disp.display(participant_ct_df.head()) return participant_ct_df ``` Result: ``` Hello: length of all_ct is 0 This is a program, but no replaced modes found. Likely cold start case. Ignoring replaced mode mapping _2023_09_default Based on 0 confirmed trips from 0 users of 0 total trips from 0 users (nan%) ``` The length of all_ct is coming up as 0, which is being passed into participant_ct_df, then eventually to expanded_ct. Since expanded_ct is zero, the blocks inside `if len(expanded_ct) > 0:` are not entered, therefore mode_counts_interest is not defined as it's being introduced inside here. There was some handling done for len(all_ct) == 0, but that didn't seem to work well, as it cascaded the issue downstream. ``` # CASE 1 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if len(all_ct) == 0: return all_ct ``` [https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867](https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867)
Tried to initialize the mode_counts_interest
outside the if condition of if len(expanded_ct)
mode_counts_interest = pd.DataFrame()
if len(expanded_ct) > 0:
This resolved the issue we're getting for Year = 2023, Month = 9.
```
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T15:06:25.626977+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Proposing this solution for the above issue:
Changes in mode_specific_timeseries:
raise RuntimeError
error.Testing the mode_specific_timeseries with moped-default mapping : PASSED
Code changes for mode_specific_timeseries:
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff viz_scripts/mode_specific_timeseries.ipynb
diff --git a/viz_scripts/mode_specific_timeseries.ipynb b/viz_scripts/mode_specific_timeseries.ipynb
index e935b48..c29bbc1 100644
--- a/viz_scripts/mode_specific_timeseries.ipynb
+++ b/viz_scripts/mode_specific_timeseries.ipynb
@@ -157,6 +157,8 @@
},
"outputs": [],
"source": [
+ "mode_counts_interest = pd.DataFrame()\n",
+ "\n",
"if len(expanded_ct) > 0:\n",
" # Get the count of unique users that were active on each given date\n",
" active_users = pd.DataFrame(data.groupby(['date_time'], as_index=False)['user_id'].nunique())\n",
@@ -180,10 +182,18 @@
" mode_counts_interest = mode_counts[mode_counts['mode_confirm']==mode_of_interest].copy()\n",
" mode_distance_interest = mode_distance[mode_distance['mode_confirm']==mode_of_interest].copy()\n",
" \n",
- " # Mapping new mode labels with dictionaries\n",
- " mode_counts['Mode_confirm'] = mode_counts['mode_confirm'].map(dic_re)\n",
- " mode_counts_interest['Mode_confirm'] = mode_counts_interest['mode_confirm'].map(dic_re)\n",
- " mode_distance_interest['Mode_confirm'] = mode_distance_interest['mode_confirm'].map(dic_re)"
+ " try:\n",
+ " if len(mode_counts_interest) == 0:\n",
+ " # force error generation so that we will go into the \"missing\" data code path\n",
+ " raise RuntimeError(f\"No {mode_of_interest} trips found\")\n",
+ "\n",
+ " # Mapping new mode labels with dictionaries\n",
+ " mode_counts['Mode_confirm'] = mode_counts['mode_confirm'].map(dic_re)\n",
+ " mode_counts_interest['Mode_confirm'] = mode_counts_interest['mode_confirm'].map(dic_re)\n",
+ " mode_distance_interest['Mode_confirm'] = mode_distance_interest['mode_confirm'].map(dic_re)\n",
+ " except:\n",
+ " print (f\"No {mode_of_interest} trips found\")\n",
+ " "
]
},
{
(END)
```
Result: Tested for moped - default mapping case
```
(emission) root@24be9fa35678:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T15:52:45.253605+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Testing the mode_specific_timeseries with moped-dynamic_config - PASSED
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.dev.yml
diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml
index fdbdbb8..c2a4342 100644
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -26,7 +26,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=dev-emulator-program
ports:
# ipynb in numbers
- "47962:8888"
```
Results:
```
(emission) root@344c3e0a7fb8:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful.
Running at 2023-09-28T16:10:27.199048+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Introduced try except block to handle raise RuntimeError error.
This is incorrect. The RuntimeError
was raised for a reason - please see the comment on the line
# force error generation so that we will go into the \"missing\" data code path\n",
" raise RuntimeError(f\"No {mode_of_interest} trips found\")\n",
Please check the blame, understand the original goal, and fix in accordance with the original vision.
wrt #89 (comment)
- Related with the page refresh one: seems to be resolved with Support custom label dropdowns from the dynamic config #89 (comment) and Support custom label dropdowns from the dynamic config #89 (comment). The metrics are being recomputed properly, we can't fix browser caching.
- Related with generic_metric_sensed: this still needs investigation
- Related with mode_specific_timeseries: I had missed this! It also still needs investigation.
@shankari Fixed the Issue (3). For issue (2) as mentioned in the above comment https://github.com/e-mission/em-public-dashboard/issues/89#issuecomment-1738494632 it's an known issue with the dataset consisting of NaN values.
There was some handling done for len(all_ct) == 0, but that didn't seem to work well, as it cascaded the issue downstream.
This is related to my previous comment. https://github.com/e-mission/em-public-dashboard/issues/89#issuecomment-1739642300
The goal is not to "get this to work somehow", it is to get it to work "the right way". Please look through the blame and address it in accordance with current functionality. In this case, the right step is to address it downstream.
an obvious fix would be to move the quality_text code into the if block in the cell above
Changes done:
Moved the quality_text code to the if block in the cell above.
Result: for mode_specific_timeseries
moped - dynamic_config:
STUDY_CONFIG=dev-emulator-program
```
(emission) root@344c3e0a7fb8:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful.
Running at 2023-09-28T17:01:41.716851+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
Test done for mode_specific_timeseries
moped - default mapping:
STUDY_CONFIG=stage-program
Result:
```
(emission) root@8c468351d76a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@8c468351d76a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
usage: generate_metrics [-h] [-d DATE DATE] plot_notebook program
generate_metrics: error: the following arguments are required: program
(emission) root@8c468351d76a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T17:14:43.888997+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
wrt #89 (comment)
- Related with the page refresh one: seems to be resolved with Support custom label dropdowns from the dynamic config #89 (comment) and Support custom label dropdowns from the dynamic config #89 (comment). The metrics are being recomputed properly, we can't fix browser caching.
- Related with generic_metric_sensed: this still needs investigation
- Related with mode_specific_timeseries: I had missed this! It also still needs investigation.
Fixed Issue (3) - moved the quality_text code to the if block in the cell above.
Tested with automated notebook execution for mode_specific_timeseries with STUDY_CONFIG=stage-program
& STUDY_CONFIG=dev-emulator-program
with moped. [showcased in above comments]
Final testing for mode_specific_timeseries
walk - default mapping:
STUDY_CONFIG=stage-program
Changes made:
Executed the following script to change mode_confirm from `moped` to `walk`
```
var result_moped = db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"moped"}, {"_id": 1})
var resultArray = result_moped.toArray();
for (var i = 0; i < resultArray.length; i++) { var doc = resultArray[i]; var docId = doc._id; var updateResult = db.Stage_analysis_timeseries.update( { "_id": docId, "data.user_input.mode_confirm": "moped" }, { "$set": { "data.user_input.mode_confirm": "walk" } } ); if (updateResult.nModified > 0) { print(`Updated document with _id: ${docId}`); } }
> db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"moped"}).count()
0
> db.Stage_analysis_timeseries.find({"data.user_input.mode_confirm":"walk"}).count()
4359
```
Results:
```
(emission) root@8c468351d76a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/stage-program.nrel-op.json
Successfully downloaded config with version 1 for Staging environment for testing programs only and data collection URL https://openpath-stage.nrel.gov/api/
Dynamic labels are not available.
Running at 2023-09-28T17:49:51.709937+00:00 with args Namespace(plot_notebook='mode_specific_timeseries.ipynb', program='default', date=None) for range (
I was double-checking the results for the energy calculations, and there is an inconsistency in https://github.com/e-mission/em-public-dashboard/pull/91#issuecomment-1715111648
In the top one, "Gas Car, with others" is 0.5. In the bottom, it is -0.0
@iantei you need to check the results carefully and make sure that they are consistent, or understand why they are inconsistent. This is the second inconsistency I have found, and I am not running the code, so I don't have the side-by-side results.
The energy calculations showcased below have different total energy of 43.03 (kWH) vs 43.53 (kWH). And we're using the dynamic_config to map for the Replaced mode in scaffolding.py.
Energy Impact (with default mapping)
Energy Impact (with dynamic_config mapping)
Most likely, due to the computation difference of energies with different mapping, this different illustration is observed.
Highlight of code change ``` # Extract translations key dic_translations = dict() if "translations" in dynamic_labels and "en" in dynamic_labels["translations"]: dic_translations = dynamic_labels["translations"]["en"] dic_translations = defaultdict(lambda: 'Other', dic_translations) # Select the mapping based on availability of dynamic_labels if dic_translations: dic_mapping = dic_translations else: dic_mapping = dic_re # Map new mode labels with translations dictionary from dynamic_labels # CASE 2 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if "mode_confirm" in expanded_ct.columns: expanded_ct['Mode_confirm'] = expanded_ct['mode_confirm'].map(dic_mapping) if study_type == 'program': # CASE 2 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if 'replaced_mode' in expanded_ct.columns: expanded_ct['Replaced_mode'] = expanded_ct['replaced_mode'].map(dic_mapping) else: print("This is a program, but no replaced modes found. Likely cold start case. Ignoring replaced mode mapping") else: print("This is a study, not expecting any replaced modes.") ```
Most likely, due to the computation difference of energies with different mapping, this different illustration is observed.
Obviously, the difference is due to the different mappings. But what is the difference and why does it cause the values to be different? We have to understand it and make sure that it is not an error. When we had the "other" trips completely dropped from (3), it was also due to a difference in the mapping, but it was an error.
most of the values here are the same - the only difference is in "Gas Car, with others" and "No Travel". Please investigate the cause of the difference between them.
@iantei for https://github.com/e-mission/em-public-dashboard/issues/89#issuecomment-1740255947, which dynamic config did you use, and how did you use it?
The dynamic config mentioned on https://github.com/e-mission/em-public-dashboard/pull/91#issuecomment-1715111648 Attached the url for the used dynamic config: example-study-label-options
@iantei
example-study-label-options
is intended for use with a study so doesn't have the replaced mode. Note that in mode_labels.csv
, there is only one set of mode key -> mode value mappings. But in the dynamic config labels, there are separate mappings for mode and replaced mode because we show them to users and there is no point in showing "No Travel" as a mode to a user, for example. You should use the replaced mode mapping to map the replaced mode values and use the program for testing.energy_intensity.csv
as the mapping. That uses the user visible value as the key. There is is a row for "Gas Car, with others"
but not one for "Gas Car Shared Ride", which is why it ends up with zero. But that is an error. Although the modes in the dynamic config are currently almost similar to the standard modes, there is no guarantee that they will be the same or even remotely similar. That's why the dynamic config has the kgCO2PerKm
value encoded as part of the config. For the dynamic config, you need to use those values instead of the values from energy_intensity.csv
.This was highlighted while creating the issue:
However, this change has not yet been implemented in the public dashboard. The public dashboard still reads the user-specific mappings and the CO2 and energy equivalents from files hardcoded in this repo (notably in viz_scripts/auxiliary_files).
The fact that the current examples are similar to the default values may be confusing you. You have to understand the underlying concepts to make sure that you build a solution that works properly.
As a concrete example, the CA e-bike rebate program is planning to support the following modes. Keep that in mind while designing and testing your solution.
Walk
Auto Driver
Auto Passenger
E-bike
Regular bike
Taxi/Uber/Lyft
Bike/Scooter-share
Bus/Train/Shuttle
Other
ERROR: Not a real trip
In the current implementation, I am using translations.en key
values create a mapping with mode_confirm
and replaced_mode
to form Mode_confirm
and Replaced_mode
respectively.
The reason behind No 'No Travel' is because there is a different in translations.en block between example-study-label-options
and example-program-label-options
. example-study-label-options.json
does not have "no_travel": "No travel". Therefore, when I chose to test with example-study-label-options, I wasn't able to see "No travel".
Choosing the dynamic_config from example-program-label-options show "No travel".
Fig: Energy_Impact chart vs Replaced mode chart with example-program-label-options
``` # Extract translations key dic_translations = dict() if "translations" in dynamic_labels and "en" in dynamic_labels["translations"]: dic_translations = dynamic_labels["translations"]["en"] dic_translations = defaultdict(lambda: 'Other', dic_translations) # Select the mapping based on availability of dynamic_labels if dic_translations: dic_mapping = dic_translations else: dic_mapping = dic_re # Map new mode labels with translations dictionary from dynamic_labels # CASE 2 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if "mode_confirm" in expanded_ct.columns: expanded_ct['Mode_confirm'] = expanded_ct['mode_confirm'].map(dic_mapping) if study_type == 'program': # CASE 2 of https://github.com/e-mission/em-public-dashboard/issues/69#issuecomment-1256835867 if 'replaced_mode' in expanded_ct.columns: expanded_ct['Replaced_mode'] = expanded_ct['replaced_mode'].map(dic_mapping) else: print("This is a program, but no replaced modes found. Likely cold start case. Ignoring replaced mode mapping") else: print("This is a study, not expecting any replaced modes.") ```
In a recent phone change, we started supporting a custom set of labels in the dropdown.
The custom set of labels is specified as a separate file linked from the dynamic config, and includes the kgCO2/km for each mode. https://github.com/e-mission/e-mission-docs/issues/945
However, this change has not yet been implemented in the public dashboard. The public dashboard still reads the user-specific mappings and the CO2 and energy equivalents from files hardcoded in this repo (notably in
viz_scripts/auxiliary_files
).This means that the custom labels will be mapped to Other, which in turn, means that the public dashboard becomes less meaningful. This is likely to be particularly challenging in international contexts - e.g. https://usaid-laos-ev-openpath.nrel.gov/public/
The public dashboard should read the values from the dynamic config as well if they are present, and fall back to defaults if they are not.
Note that depending on how we convert between energy and emissions, we may have to have the energy equivalent stored in the trip_confirm lists as well. You can submit a PR (similar to https://github.com/e-mission/nrel-openpath-deploy-configs/pull/32) to add them if required.
@ananta-nrel can you please handle this?