shankari
commented
9 months ago @iantei can you please address this?
Closed shankari closed 8 months ago
shankari
commented
9 months ago @iantei can you please address this?
iantei
commented
9 months ago Yes, I will fix this.
shankari
commented
9 months ago I couldn't wait any longer for Laos (they have been collecting data for two weeks and wanted a check-in today), so I pushed this to production for the Lao EV project
And it looks like it works, at least for the months. Still showing old values for overall, but hopefully that is just a caching issue. I don't see any other errors in the logs.
| Dec 2024 | All data |
|---|---|
shankari
commented
9 months ago @iantei Please also test against staging and the USAID Laos project to see if there are any other showstoppers before we push out to production in general.
shankari
commented
9 months ago Ah the "All Data" has now been updated 🎉
However, @iantei unlike the previous implementation, the colors are not consistent between graphs, which makes it a bit confusing while comparing - i.e. "Dark Blue" = Motorcycle in trip count versus "Dark Blue" = Airplane in trip distance.
If you have the stacked bar chart implementation ready, and it fixes this, I am OK with fixing it then. But if you don't have that yet, we may need to fix it sooner. Comments?
iantei
commented
9 months ago Confirmed using https://openpath-stage.nrel.gov/public/?study_config=smart-commute-ebike which does not have the energy impact calculation
Completed the changes for the STUDY_CONFIG=smart-commute-ebike. Now it'll reflect the energy impact calculation option as a metric dropdown list.
| Smart Commute eBike Program Public Dashboard |
|---|
I am facing some issue while trying to push the changes [Cloned the repo again, created branch, tried to push changes.],
ashrest2-35384s:em-public-dashboard ashrest2$ git push --set-upstream origin Fix_Same_Metrics_For_Dynamic_and_Default_Mapping
remote: Permission to e-mission/em-public-dashboard.git denied to iantei.
fatal: unable to access 'https://github.com/e-mission/em-public-dashboard/': The requested URL returned error: 403
remote: Permission to e-mission/em-public-dashboard.git denied to iantei.
fatal: unable to access 'https://github.com/e-mission/em-public-dashboard/': The requested URL returned error: 403I am trying to fix this. I regenerated the Personal Access Token, and tried to use it using the below command. Still getting the same error.
git remote set-url origin https://iantei:<Personal Access Token>@github.com/e-mission/em-public-dashboard
Meanwhile, I will some further testing with staging and USAID Laos Project too.
iantei
commented
9 months ago Ah the "All Data" has now been updated 🎉
However, @iantei unlike the previous implementation, the colors are not consistent between graphs, which makes it a bit confusing while comparing - i.e. "Dark Blue" = Motorcycle in trip count versus "Dark Blue" = Airplane in trip distance.
If you have the stacked bar chart implementation ready, and it fixes this, I am OK with fixing it then. But if you don't have that yet, we may need to fix it sooner. Comments?
The stacked bar chart implementation in
plot_canbikeco.pyis good to experiment with, but I am not ready to integrate this withgeneric_metricsto replace the existing pie-chart yet. Meanwhile, I can look into incorporating this color association with Mode_confirm - should have been handled in previous implementation which has been changed now.
shankari
commented
9 months ago @iantei
remote: Permission to e-mission/em-public-dashboard.git denied to iantei.
we use fork + pull request. you don't have access to the upstream fork
iantei
commented
9 months ago I observe different behaviors for the STUDY_CONFIG=usaid-loas-ev in stage vs development testing.
Process followed to generate the development mode STUDY_CONFIG=usaid-laos-ev:
Code changes:
```
- CRON_MODE=
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=usaid-laos-ev
ports:
# ipynb in numbers
- "47962:8888"
```
Execution steps:
```
(emission) root@c80c308627f2:/usr/src/app/saved-notebooks# PYTHON=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@c80c308627f2:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2023-12-05T07:14:27.268219+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Even though usaid-laos-ev study config has dynamic config available i.e. label_options is available, it shows as the default mapping mode in the Staging environment. That is different from the development testing environment.
I am not sure what could be resulting in this issue.
iantei
commented
9 months ago @iantei can you please address this?
I have fixed the changes for this issue here. #104
I am keeping this open with the consideration, we might want to test, observe any other required changes and deliver the fix changes in this PR itself.
shankari
commented
9 months ago @iantei please put the PR into "Ready for review"
wrt differences between
Staging: https://openpath-stage.nrel.gov/public/?study_config=usaid-laos-ev Development: http://localhost:3274/?study_config=usaid-laos-ev
the study_config GET param is only used for the frontend display (think about it - how would you pass the information from the frontend to the viz_scripts?). So the actual images will still be from the staging config.
As I said in https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1839956624
I couldn't wait any longer for Laos (they have been collecting data for two weeks and wanted a check-in today), so I pushed this to production for the Lao EV project
The Laos production environment is at: https://usaid-laos-ev-openpath.nrel.gov/public/
If we had more time before it was pushed to production, we would have been able to take better before and after snapshots, but the code was not ready until the last minute.
You can use the snapshots above (for the trip count before and after the change) for the comparison.
iantei
commented
9 months ago @iantei please put the PR into "Ready for review"
I have moved the PR into "Ready for Review" stage. #104
The Laos production environment is at: https://usaid-laos-ev-openpath.nrel.gov/public/
Things look good at the Laos production environment.
iantei
commented
9 months ago Launched the following site [https://usaid-laos-ev-openpath.nrel.gov/public/](https://usaid-laos-ev-openpath.nrel.gov/public/) Loaded all the charts with `Month = All data` For the aggregated data, I have enlisted all the charts. | Laos Production Charts (Aggregate) | |--------| | |
Launched the following site [https://usaid-laos-ev-openpath.nrel.gov/public/](https://usaid-laos-ev-openpath.nrel.gov/public/) Loaded all the charts with `Month = 11/2023` For the aggregated data, I have enlisted all the charts. | Laos Production Chart (11/2023) | |--------| |  |
Common observations in both the cases above:
Average trip length (selected by users) All data Default and Average trip length (sensed) All data Default."Timeseries of emissions per kilometer""Timeseries of emissions"displays the distance/weight in Metric system (as the Laos config has dynamic config) i.e. kg/km
while the following other charts like:
"Trip count under 10 miles", "Trip count under 10 miles (sensed) All data", "Trip miles by mode All data Default", "Trip miles by mode (sensed) All Data Default", "Trip miles by land mode (sensed) All data Default", "Average trip length All data Default" still uses Imperial system (existing implementation), depicting distance as in milesCould this be a cause of concern or confusion to the end user?
shankari
commented
9 months ago There are no charts available for Average trip length (selected by users) All data Default and Average trip length (sensed) All data Default.
Interesting. Is this also true on staging (with the default study_config? Let's try to reproduce and see if there was a regression.
As mentioned above https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1839978693 there not a fixed association of color palette with the Mode_confirm. This might lead to some confusion to the end-user.
Correct, we need to fix this, either immediately, or (if the stacked bar charts are imminent), then while implementing them.
while the following other charts like: .... still uses Imperial system (existing implementation), depicting distance as in miles Could this be a cause of concern or confusion to the end user?
Yes. There are many aspects of the public dashboard that need to respect the dynamic config. The first and most important one was the mode_of_interest (implemented much earlier). The second was the dynamic labels, which we finished just barely in time (I had deploy to Laos without waiting for the full testing on staging). IIRC, we discussed using imperial units earlier, but decided to defer to focus on higher priority dynamic label change.
https://github.com/e-mission/em-public-dashboard/issues/89#issuecomment-1783577702
Per the comment, have you already filed an issue? If so, we should track the useImperial change there but link it here for context.
iantei
commented
9 months ago
Code changes:
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73a9875..9e495ab 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -21,7 +21,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=smart-commute-ebike
ports:
# ipynb in numbers
- "47962:8888"
```
Steps followed:
1. Spin up the `docker-compose`
```
docker-compose -f docker-compose.dev.yml up
```
2. Launch bash for `em-public-dashboard-notebook-server`
In the next terminal to execute generate plots for different notebooks:
```
docker exec -it em-public-dashboard-notebook-server-1 /bin/bash
```
`Generic_Metrics` notebook:
```
(emission) root@43a25923608a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@43a25923608a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/smart-commute-ebike.nrel-op.json
Successfully downloaded config with version 1 for Smart Commute eBike Program and data collection URL https://smart-commute-ebike-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in smart-commute-ebike
Running at 2023-12-06T17:46:59.268655+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Results:
| Charts for `study_config` = `smart-commute-ebike` |
|--------|
|  |
3. Execution for different notebooks
Observations:
There is a slight issue with the arrangement of charts like "Timeseries of proposition ofe-bikespecific miles All data Default", which was immobile and was placed on top of the existing charts.
Illustration of the overlapping charts issue:
There is a work-around for this issue though. Close those overlapping charts and re-launch it.
| Possible work around solution: Results: | After following possible work-around solution |
|---|---|
We can observe no charts are available for Average trip length (selected by users) All data Default even with this study_config = smart-commute-ebike.
iantei
commented
9 months ago Execution of Smart Commute E-bike: Working on with the same charts generated from earlier iteration for Smart Commute E-bike.
Using dataset: vail dataset
| Some observation: | Smart Commute eBike Program Public Dashboard |
|---|---|
Each the individual month selection were not able to generate charts, while the aggregate one was able to give the chart representation properly. Isn't it required that at least one of the individual chart should have enough data to generate the chart alongside aggregate one?
Observed the same for the below metrics too:
- Number of Trips
- Number of commute trips
- Trip count by purpose
- Trip count under 10 milesAvailable dates: 3/2023 - 12/2023 Tested on Chrome.
shankari
commented
9 months ago @iantei when you are checking the frontend and not the graphs, is there a reason you are not testing on staging directly? You are still using the same dataset, so if there are errors in working with the data, they will (largely) apply across all configs. Can you please understand the interplay between frontend config, viz scripts config, container type and dataset and come up with a testing plan?
iantei
commented
9 months ago Ah the "All Data" has now been updated 🎉
The below test have been conducted after launching: https://usaid-laos-ev-openpath.nrel.gov/public/
In reference to this comment's status about the chart representation in Laos
| Previous | Latest (06/Dec/2023) |
|---|---|
It's not possible to compare between these two charts distinctly because the total number of testers has changed from 6 to 10 and 19 to 20 respectively for 12/2023 charts and All Data charts.
Similarly, the participants's total trips and number of users have also changed.
| One of the observation in the below chart is: [Observed on 6 Dec 2023] | Number of trips 12/2023 |
|---|---|
Even though the distribution of Other is higher (7.8% (6)) than Walk (6.5% (5)), it is shows in the later pie order, which doesn't look right, accounting all distribution of charts' pie are happening in anti-clockwise order in increasing distribution.
iantei
commented
9 months ago @iantei when you are checking the frontend and not the graphs, is there a reason you are not testing on staging directly? You are still using the same dataset, so if there are errors in working with the data, they will (largely) apply across all configs. Can you please understand the interplay between frontend config, viz scripts config, container type and dataset and come up with a testing plan?
According to the meeting discussion we had yesterday, I understood that I do not have access to staging, dataset or configuration changes related with staging.
So, I would have to make change on the docker-compose environment for study_config for docker.compose.yml, generate the charts and observe the changes via. access to the http://localhost:3274/?study_config=smart-commute-ebike
I have scheduled a meeting with you. It would be great if we can clarify it over the meeting.
shankari
commented
9 months ago @iantei given that we did discuss it yesterday and it wasn't clear, can you please add in what your understanding and proposal is, and I can respond in writing? That might make it easier to have a clear plan going forward.
iantei
commented
9 months ago Sure. Let me explain you my understanding first. Please correct me accordingly.
viz_scripts. The first one is docker-compose.dev.yml and the second one is docker-compose.yml.
For development testing:
I would launch docker-compose.dev.yml with the required study_config in the environment section of it.
In the next terminal, launch docker exec -it em-public-dashboard-notebook-server-1 /bin/bash
Execute the following scripts:
PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
...This will load the corresponding charts file in the /plots folder.
Then correspondingly, the frontend would pick up these plots to showcase on the dashboard, according to the selection of options.
My understanding is, since I do not have access to staging environment configuration changes. I would try to emulate it by spinning up the docker-compose.yml instead itself.
@iantei when you are checking the frontend and not the graphs, is there a reason you are not testing on staging directly?
My understanding is I do not have access to the staging configuration changes or the dataset used in staging environment. I tried the following,
You are still using the same dataset, so if there are errors in working with the data, they will (largely) apply across all configs.
For this part - the vail dataset is only giving issues with the sensed related chart representation. In this case, the charts are being generated well for the aggregated data while none of the individual ones are being generated.
My concern was - how could it generate the chart well for aggregated data while none of the individual charts are generated properly?
Can you please understand the interplay between frontend config, viz scripts config, container type and dataset and come up with a testing plan?
Here's my understanding about this.
I had initially loaded the vail dataset, with which I am conducting the test in development mode. There are three services specified in the docker-compose file:
I have been using the vail dataset, which I have loaded and stored in the db and is being used by the viz_scripts to generate the charts, which would in turn be used by frontend to display charts according to metrics selected on the dashboard.
When you asked me to test against staging, I wasn't sure about proceed to test in staging environment, and felt that emulating the development testing scenario but with docker-compose.yml spin up could be equivalent.
Could you please let me know the appropriate way to "test against staging"?
shankari
commented
9 months ago @iantei there are three aspects that come together in the public dashboard:
(1) is public for all studies and all programs (2) we can pull datasets as needed (vail is not a canonical dataset) (3) you can launch containers locally, and access public interfaces of containers in the cloud
You need to think through what you are testing, and how you can assemble the pieces to check that it works.
As a concrete example and an explanation for "test against staging", consider testing the frontend. You can access http://localhost:... by running "prod" containers on your laptop, but that is not really different from accessing the dev frontend. Note that the dev and prod Dockerfiles are not substantially different for the frontend (only the exposed port is different).
For the frontend, you want to check that the cloud deployment is correct since that is the big difference. So you should test using https://openpath-stage.nrel.gov/public which you can access (as can anybody else in the world).
When running the viz_scripts, the difference between dev and production is in other areas, and you should focus on testing those. Hope that helps!
I would suggest thinking through what those differences are, and coming up with a plan to test them. This may involve additional support from me.
iantei
commented
9 months ago For the frontend, you want to check that the cloud deployment is correct since that is the big difference. So you should test using https://openpath-stage.nrel.gov/public which you can access (as can anybody else in the world).
The https://openpath-stage.nrel.gov/public/ takes the study_config as stage-study as default.
When I change the study_config to usaid-laos-ev and load https://openpath-stage.nrel.gov/public/?study_config=usaid-laos-ev, I am unable to see the changes in the charts since the viz_scripts are not executed with this changed study_config param.
For the development testing in localhost, I change the environment in the docker-compose.dev.yml and/or docker-compose.yml for study_config to the required one and load the frontend with the updated config param.
In this way:
Changes in the docker-compose.dev.yml
notebook-server:
...
environment:
...
- STUDY_CONFIG=smart-commute-ebikeRe-generate all the charts by getting inside the public-dashboard-notebook-server.
Testing these changes:
Launch: http://localhost:<3274>/?study_config=smart-commute-ebike
For the staging testing, is there any specific container (other than docker-compose.dev.yml or docker-compose.yml ), which I can modify to integrate the change of config, so it gets reflected on the https://openpath-stage.nrel.gov/public/?study_config=smart-commute-ebike? Basically, I want to know the process of using different config to test in staging environment.
shankari
commented
9 months ago Your comment in https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1846206725 is focused on testing viz_scripts. My comment in https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1843623373 talked about the frontend and not the graphs.
when you are checking the frontend and not the graphs, is there a reason you are not testing on staging directly?
I repeated this in https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1845917856
For the frontend, you want to check that the cloud deployment is correct since that is the big difference. So you should test using https://openpath-stage.nrel.gov/public which you can access (as can anybody else in the world).
You need to test both the frontend and the viz_scripts and you need to do test them in different ways because the meaningful differences between dev and staging are different for the two components.
For the staging testing, is there any specific container ... which I can modify to integrate the change of config, so it gets reflected on the https://openpath-stage.nrel.gov/public/?study_config=smart-commute-ebike?
No. How can there be?
Other comments:
The https://usaid-laos-ev-openpath.nrel.gov/public/ takes the study_config as stage-study as default.
This is incorrect. This takes usaid-laos-ev as the study_config by default.
(other than docker-compose.dev.yml or docker-compose.dev.yml ),
Not sure what you mean here - these are the same file.
iantei
commented
9 months ago Other comments:
The https://usaid-laos-ev-openpath.nrel.gov/public/ takes the study_config as stage-study as default.
This is incorrect. This takes
usaid-laos-evas thestudy_configby default.
Yes, that's right. Updated the above comment - I used the wrong text representation for the right hyperlink. I meant for https://openpath-stage.nrel.gov/public
(other than docker-compose.dev.yml or docker-compose.dev.yml ),
Not sure what you mean here - these are the same file.
Corrected the above comment: I meant, docker-compose.dev.yml or docker-compose.yml instead.
iantei
commented
9 months ago For the staging testing, is there any specific container ... which I can modify to integrate the change of config, so it gets reflected on the https://openpath-stage.nrel.gov/public/?study_config=smart-commute-ebike?
No. How can there be?
I have tested the viz_scripts execution in docker-compose.yml already #103(comment). My question to look up for different container was to understand if are ways to validate for charts in the staging environment for different configurations.
iantei
commented
9 months ago Based on the above discussion, Here's my high level test plan to test the changes on staging before pushing to production. Please let me know if any revision in the approach is required.
There are two aspects we need to check for:
Frontend: [Testing in both localhost, and staging environment] There is either a Study or Program with or without dynamic labels. A. For Study - a. Are all the default charts being displayed? b. Are all Metrics being listed for selection? c. Are all these metrics generating the charts [The values of the chart might not depict the correct values]? B. For Program - a. Are all the default charts being displayed? b. Are all Metrics being listed for selection? c. Are all these metrics generating the charts [The values of the chart might not depict the correct values]?
Viz_scripts: [Test with both dev and prod docker compose, and load into localhost] Can be deployed in either production environment or development environment
docker-compose.yml vs docker-compose.dev.ymllocalhost and further validated.Test both the aspects above with the below combination:
This way, we're validating all the default and list of metrics to be displayed for both program/study with/out dynamic labels in the em-public-dashboard.
Secondly, we're validating the charts are being generated or not in both development and production environment for all notebooks.
We are already validating the charts generated in prod environment through localhost. Still - I am not sure, I have a question. How can we test the charts (not frontend) in staging environment with different configurations. Could you please help me understand this?
Note: The Laos config has already been tested in the production environment #103(comment)
iantei
commented
9 months ago NOTE: Still updating
Frontend Testing:
Process of testing:
Launch the following sites:
And test for the below configurations, which are presented as columns below -
| Index | Config Type | Are all the default charts being displayed? | Are all metrics being listed for selection? | Are all metrics generating the charts? |
|---|---|---|---|---|
| 1 | Default Mapping + Study washingtoncommons | Yes (7/7) | Yes (21/21) |
No (Average Trip Length (sensed), Average Trip Length (selected by users) missing) |
| 2 | Default Mapping + Program ebikegj | Yes (7/7) | No (33/35) |
No (Average Trip Length (sensed), Average Trip Length (selected by users) missing) |
| 3 | Dynamic Config + Study usaid-laos-ev | Yes (7/7) | Yes (20/20) |
No (Average Trip Length (sensed), Average Trip Length (selected by users) missing) |
| 4 | Dynamic Config + Program dev-emulator-program | Yes (7/7) | Yes (33/33) |
No (Average Trip Length (sensed), Average Trip Length (selected by users) missing) |
Observations:
e-bike specific Energy Impact".shankari
commented
9 months ago We are already validating the charts generated in prod environment through localhost. Still - I am not sure, I have a question. How can we test the charts (not frontend) in staging environment with different configurations. Could you please help me understand this?
We cannot [1]. Pop quiz: why can't we?
Let's not lose the forest for the trees. Our overall goal is to ensure that when we go to production, it will not break. Staging offers a production-like environment to help ensure that. So while planning testing, you need to think about what is different between dev and prod, and how you can test those differences before pushing to prod.
[1] That is a very blanket statement. Of course, we can do many things if we had lots of time and resources. But we cannot do so without completely changing our current staging setup in ways that will cause it to diverge from
All these metrics are not able to generate charts for "Average Trip Length (sensed)" and "Average Trip Length (selected by users) missing".
For Default Mapping + Program, there are missing selection options for "Timeseries of energy" and "Sketch of e-bike specific Energy Impact".
isn't this what was originally reported and fixed in https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1840113382
iantei
commented
9 months ago For Default Mapping + Program, there are missing selection options for "Timeseries of energy" and "Sketch of e-bike specific Energy Impact".
isn't this what was originally reported and fixed in #103 (comment)
Yes, this issue is the same as one originally reported. I had validated this against the http://localhost:3274/?study_config=smart-commute-ebike. The changes have been merged into the main branch too. However, I couldn't observe these changes being reflected on stage environment while testing with https://openpath-stage.nrel.gov/public/?study_config=ebikegj which is the same combination of program + default mapping.
iantei
commented
9 months ago We are already validating the charts generated in prod environment through localhost. Still - I am not sure, I have a question. How can we test the charts (not frontend) in staging environment with different configurations. Could you please help me understand this?
We cannot [1]. Pop quiz: why can't we?
From my understanding, There is a combination of generating static charts on the basis of the configuration we select. And there is a frontend part which is responsible to show all these generated static charts, upon the selection of different metrics and months option from the dropdown menu.
Currently, https://openpath-stage.nrel.gov/public/ only showcases charts for the config - stage-study. In order to accommodate for all the different configurations, we would need to generate these charts on the backend, and periodically update the charts. Confining to this limitation, we cannot simply test with different chart of different configurations.
The workaround for this - Simulating the production environment, where the charts are being generated with different configurations. Execution of docker-compose.yml with the specific STUDY_CONFIG and testing these charts on localhost:3274 would help replicate the production environment.
all of these are on staging, so all the figures will be the same. I am not sure what you get by the "Are all metrics generating the charts?" column. You need to test that differently.
For the above test scenario, I tested whether the selection of the metric option would launch the chart or not.
Yes, I will be testing the scenario of generating the charts with running the docker-compose.yml as explained above.
iantei
commented
9 months ago
STUDY_CONFIG: `washingtoncommonsj`
Changes in the code:
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73a9875..27a0851 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -21,7 +21,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=washingtoncommons
ports:
# ipynb in numbers
- "47962:8888"
```
Execution processs:
```
(emission) root@8676bfe8687e:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@8676bfe8687e:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/washingtoncommons.nrel-op.json
Successfully downloaded config with version 1 for Impact of Collaborative Neighborhoods on Vehicle Miles Traveled and data collection URL https://washingtoncommons-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in washingtoncommons
Running at 2023-12-11T17:55:23.135790+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
STUDY_CONFIG: `ebikegj`
Code changes:
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73a9875..3a4adb1 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -21,7 +21,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=ebikegj
ports:
# ipynb in numbers
- "47962:8888"
```
Execution Process:
```
(emission) root@0c7cbcb1520c:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@0c7cbcb1520c:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/ebikegj.nrel-op.json
Successfully downloaded config with version 1 for eBike to Work Ownership Program and data collection URL https://ebikegj-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in ebikegj
Running at 2023-12-11T18:08:28.861092+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
STUDY_CONFIG: dev-emulator-program
Code changes:
```
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73a9875..b250627 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -21,7 +21,7 @@ services:
- DB_HOST=db
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=dev-emulator-program
ports:
# ipynb in numbers
- "47962:8888"
```
Execution process:
```
(emission) root@36598ba6773a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@36598ba6773a:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful for nrel-openpath-deploy-configs: dev-emulator-program
Running at 2023-12-11T18:32:53.535711+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (Changes for `STUDY_CONFIG:washingtoncommons`
Changes : Default mapping + program - ebikegj
The test for Dynamic Config + Study has already been tested with the [Issue #103 Comment](https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1841545398)
Changes for Dynamic config + program: dev-emulator-program
iantei
commented
9 months ago Why are "Average Trip Length (selected by users)" and "Average Trip Length (sensed)" not generating any charts? [Investigating]
At first, the values associated with these charts - i.e. average_miles_mode_confirm2 and average_miles_sensed_mode which are supposed to be associated with file_name in notebooks, are not available in any of the notebooks.
iantei
commented
9 months ago "Average Trip Length (selected by users)" "Average Trip Length (sensed)" are also missing from the production public dashboard related with the https://ebikegj-openpath.nrel.gov/public/ E-bike to work Ownership program.
shankari
commented
9 months ago all of these are on staging, so all the figures will be the same. I am not sure what you get by the "Are all metrics generating the charts?" column. You need to test that differently.
Yes, I will be testing the scenario of generating the charts with running the docker-compose.yml as explained above.
Please read through https://github.com/e-mission/em-public-dashboard/issues/103#issuecomment-1845917856 carefully
Which of the three aspects does your proposed viz_script testing strategy tackle?
iantei
commented
9 months ago I have the following understanding about the missing charts above:
"Average Trip Length (selected by users)"The generation of chart associated with "Average Trip Length (selected by users)" have been removed in the commit #51 . Further, I do not see any file_name associated with average_miles_mode_confirm2 in the current main branch.
Possible solution:
There's barplot_mode2() present on plots.py. We could revert back this change, and it would likely work.
"Average Trip Length (sensed)"I do not see any changes which would generate the file average_miles_sensed_mode, while all other sensed related metric chart files are being generated here. #92
Possible solution:
We will need to incorporate changes for the average_miles_sensed_mode in the generic_metrics_sensed.ipynb.
Used the following command to identify the last commits consisting of these keywords:
ashrest2-35384s:em-public-dashboard ashrest2$ git log --oneline --all -S'average_miles_mode_confirm2' -- '*.ipynb'
85a4cce Use database value for mode
28449df Fixes from the production code
ashrest2-35384s:em-public-dashboard ashrest2$ git log --oneline --all -S'average_miles_sensed_mode' -- '*.ipynb'
ashrest2-35384s:em-public-dashboard ashrest2$ git log --oneline --all -S'ntrips_sensed_per_day' -- '*.ipynb'
8903b11 ✨ add the `sensed` prefix to the metrics generated from sensed dataThen looked up for these commits to observe the changes.
Note: These observations are not the collateral of the deliverables from #91 since it doesn't have changes related with these.
iantei
commented
9 months ago all of these are on staging, so all the figures will be the same. I am not sure what you get by the "Are all metrics generating the charts?" column. You need to test that differently.
Yes, I will be testing the scenario of generating the charts with running the docker-compose.yml as explained above.
Please read through #103 (comment) carefully Which of the three aspects does your proposed
viz_scripttesting strategy tackle?
I have tested the public dashboard functionality like the default metrics loading, and metrics options list and the metrics options selection loading a chart with the Frontend testing.
With the viz_script, I want to cover the part for the production docker container where all the charts are being generated well for different study/ program with default mapping/dynamic config setting.
I think my viz_scripts testing strategy tests for the 1st aspect of testing against different "study/program configuration" alongside the third aspect of testing with prod container over the dev container.
I didn't understand the idea of testing the publicly available interfaces from the locally run container. I understand there is two approaches to do the test of charts:
iantei
commented
9 months ago Testing with different variations of dataset in prod container.
```
> show dbs
Stage_database 0.000GB
admin 0.000GB
config 0.000GB
local 0.000GB
```
```
(emission) root@b6cde2031614:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@b6cde2031614:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/washingtoncommons.nrel-op.json
Successfully downloaded config with version 1 for Impact of Collaborative Neighborhoods on Vehicle Miles Traveled and data collection URL https://washingtoncommons-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in washingtoncommons
Running at 2023-12-14T09:46:19.623763+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
```
(emission) root@30250175f3cc:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@30250175f3cc:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/ebikegj.nrel-op.json
Successfully downloaded config with version 1 for eBike to Work Ownership Program and data collection URL https://ebikegj-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in ebikegj
Running at 2023-12-14T10:12:26.022948+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
```
(emission) root@cef28cd6ee37:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
Running at 2023-12-14T10:29:28.501570+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={'MODE': [{'value': 'walk', 'baseMode': 'WALKING', 'met_equivalent': 'WALKING', 'kgCo2PerKm': 0}, {'value': 'e-bike', 'baseMode': 'E_BIKE', 'met': {'ALL': {'range': [0, -1], 'mets': 4.9}}, 'kgCo2PerKm': 0.00728}, {'value': 'bike', 'baseMode': 'BICYCLING', 'met_equivalent': 'BICYCLING', 'kgCo2PerKm': 0}, {'value': 'bikeshare', 'baseMode': 'BICYCLING', 'met_equivalent': 'BICYCLING', 'kgCo2PerKm': 0}, {'value': 'scootershare', 'baseMode': 'E_SCOOTER', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.00894}, {'value': 'drove_alone', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.22031}, {'value': 'shared_ride', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.11015}, {'value': 'e_car_drove_alone', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.08216}, {'value': 'e_car_shared_ride', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.04108}, {'value': 'moped', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.05555}, {'value': 'taxi', 'baseMode': 'TAXI', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.30741}, {'value': 'bus', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'train', 'baseMode': 'TRAIN', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.12256}, {'value': 'free_shuttle', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'air', 'baseMode': 'AIR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.09975}, {'value': 'not_a_trip', 'baseMode': 'UNKNOWN', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}, {'value': 'other', 'baseMode': 'OTHER', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}], 'PURPOSE': [{'value': 'home'}, {'value': 'work'}, {'value': 'at_work'}, {'value': 'school'}, {'value': 'transit_transfer'}, {'value': 'shopping'}, {'value': 'meal'}, {'value': 'pick_drop_person'}, {'value': 'pick_drop_item'}, {'value': 'personal_med'}, {'value': 'access_recreation'}, {'value': 'exercise'}, {'value': 'entertainment'}, {'value': 'religious'}, {'value': 'other'}], 'REPLACED_MODE': [{'value': 'no_travel'}, {'value': 'walk'}, {'value': 'bike'}, {'value': 'bikeshare'}, {'value': 'scootershare'}, {'value': 'drove_alone'}, {'value': 'shared_ride'}, {'value': 'e_car_drove_alone'}, {'value': 'e_car_shared_ride'}, {'value': 'taxi'}, {'value': 'bus'}, {'value': 'train'}, {'value': 'free_shuttle'}, {'value': 'other'}], 'translations': {'en': {'walk': 'Walk', 'e-bike': 'E-bike', 'bike': 'Regular Bike', 'bikeshare': 'Bikeshare', 'scootershare': 'Scooter share', 'drove_alone': 'Gas Car Drove Alone', 'shared_ride': 'Gas Car Shared Ride', 'e_car_drove_alone': 'E-Car Drove Alone', 'e_car_shared_ride': 'E-Car Shared Ride', 'moped': 'Moped', 'taxi': 'Taxi/Uber/Lyft', 'bus': 'Bus', 'train': 'Train', 'free_shuttle': 'Free Shuttle', 'air': 'Air', 'not_a_trip': 'Not a trip', 'no_travel': 'No travel', 'home': 'Home', 'work': 'To Work', 'at_work': 'At Work', 'school': 'School', 'transit_transfer': 'Transit transfer', 'shopping': 'Shopping', 'meal': 'Meal', 'pick_drop_person': 'Pick-up/ Drop off Person', 'pick_drop_item': 'Pick-up/ Drop off Item', 'personal_med': 'Personal/ Medical', 'access_recreation': 'Access Recreation', 'exercise': 'Recreation/ Exercise', 'entertainment': 'Entertainment/ Social', 'religious': 'Religious', 'other': 'Other'}, 'es': {'walk': 'Caminando', 'e-bike': 'e-bicicleta', 'bike': 'Bicicleta', 'bikeshare': 'Bicicleta compartida', 'scootershare': 'Motoneta compartida', 'drove_alone': 'Coche de Gas, Condujo solo', 'shared_ride': 'Coche de Gas, Condujo con otros', 'e_car_drove_alone': 'e-coche, Condujo solo', 'e_car_shared_ride': 'e-coche, Condujo con ontras', 'moped': 'Ciclomotor', 'taxi': 'Taxi/Uber/Lyft', 'bus': 'Autobús', 'train': 'Tren', 'free_shuttle': 'Colectivo gratuito', 'air': 'Avión', 'not_a_trip': 'No es un viaje', 'no_travel': 'No viajar', 'home': 'Inicio', 'work': 'Trabajo', 'at_work': 'En el trabajo', 'school': 'Escuela', 'transit_transfer': 'Transbordo', 'shopping': 'Compras', 'meal': 'Comida', 'pick_drop_person': 'Recoger/ Entregar Individuo', 'pick_drop_item': 'Recoger/ Entregar Objeto', 'personal_med': 'Personal/ Médico', 'access_recreation': 'Acceder a Recreación', 'exercise': 'Recreación/ Ejercicio', 'entertainment': 'Entretenimiento/ Social', 'religious': 'Religioso', 'other': 'Otros'}}})]
(emission) root@cef28cd6ee37:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful for nrel-openpath-deploy-configs: dev-emulator-program
Running at 2023-12-14T10:34:39.727942+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (Execution For STUDY_CONFIG = `washingtoncommons`
Runs successfully for all notebooks
Execution For STUDY_CONFIG = `ebikegj`
Runs successfully for all notebooks
Execution For STUDY_CONFIG = `dev-emulator-program`
Runs successfully for all notebooks
Changes:
```
ashrest2-35384s:em-public-dashboard ashrest2$ bash viz_scripts/docker/load_mongodump.sh fc_2022-05-09.tar.gz
2023-12-14T10:46:58.329+0000 2562838 document(s) restored successfully. 0 document(s) failed to restore.
ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-db-1 mongo
> show dbs
Stage_database 1.109GB
admin 0.000GB
config 0.000GB
local 0.000GB
```
```
(emission) root@cef28cd6ee37:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/dev-emulator-program.nrel-op.json
Successfully downloaded config with version 1 for Development environment (program) and data collection URL default
Dynamic labels download was successful for nrel-openpath-deploy-configs: dev-emulator-program
Running at 2023-12-14T10:54:22.006452+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (Execution For STUDY_CONFIG = `dev-emulator-program`
Runs successfully for all notebooks EXCEPT `generic_metrics_sensed.ipynb` notebook
iantei
commented
8 months ago @shankari Issue observed while testing with the dataset for usaid-laos-ev:
Procedure followed:
docker-compose.yml 's STUDY_CONFIG=usaid-laos-ev> show dbs
admin 0.000GB
config 0.000GB
local 0.000GBusaid-laos-ev for 20th Dec 2023.> show dbs
Stage_database 4.409GB
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.761GBExecuted the following:
Executed for generic_metrics and generic_timeseries
Couldn't launch for mode_specific_metrics, mode_specific_timeseries and energy_calculations with the following error:
----- stderr -----
Exception: The plots in this notebook are only relevant to programs
------------------
```
(emission) root@d66e8c456113:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@d66e8c456113:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2023-12-21T01:43:03.426453+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
| http://localhost:3274/?study_config=usaid-laos-ev | https://usaid-laos-ev-openpath.nrel.gov/public/ |
|---|---|
| Total Confirmed Trips: 7856 , Participants: 19954, Testers: 29 | Total Confirmed Trips: 599 , Participants: 5253, Testers: 20 |
There seems to be a difference in data between these charts, and thereby all other charts are also different.
Am I doing something wrong in the above process?
shankari
commented
8 months ago Couldn't launch for mode_specific_metrics, mode_specific_timeseries and energy_calculations with the following error:
Exception: The plots in this notebook are only relevant to programs
As you can see from the exception, this will happen for all studies. Did you not see this in WashingtonCommons?
Total Confirmed Trips: 7856 , Participants: 19954, Testers: 29
This is not what the graph is showing. The one on the left does not have ~ 20k participants.
Re-loaded the dataset associated with usaid-laos-ev for 20th Dec 2023.
Looks like it has both stage data and laos data loaded. Are you sure you are reading the correct one?
> show dbs
Stage_database 4.409GB
openpath_prod_usaid_laos_ev 0.761GBCouldn't launch for mode_specific_metrics, mode_specific_timeseries and energy_calculations with the following error:
Exception: The plots in this notebook are only relevant to programsAs you can see from the exception, this will happen for all studies. Did you not see this in WashingtonCommons?
Yes, this is well expected with all studies. I did observe this with other studies too. I just wanted to make a note for it here.
Total Confirmed Trips: 7856 , Participants: 19954, Testers: 29
This is not what the graph is showing. The one on the left does not have ~ 20k participants.
Left Chart: The chart represents total confirmed trips: 2782 + 2052 + 1500 + 1065 + 457 = 7856 (Total Confirmed Trips) Right Chart: The chart represents total confirmed trips: 348 + 111 + 70 + 47 = 576 + Car Shared Ride (23) =~ 599. The share for Car Shared Ride is less ~ 3.84% therefore it is not displayed on the chart. The ~20K is the Total Trips (referred from text above the chart).
Re-loaded the dataset associated with usaid-laos-ev for 20th Dec 2023.
Looks like it has both stage data and laos data loaded. Are you sure you are reading the correct one?
> show dbs Stage_database 4.409GB openpath_prod_usaid_laos_ev 0.761GB
Yes, I removed the existing dataset, and loaded the dataset which you've shared with me.
Dropping the Stage_database leads to error in generation of charts "Unable to generate plot Number of trips for each mode (selected by users) Reason:".
I am placing the openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gz file inside em-public-dashboard folder, and executing the following command to load the dataset.
bash viz_scripts/docker/load_mongodump.sh openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gzTotal Confirmed Trips: 7856 , Participants: 19954, Testers: 29 This is not what the graph is showing. The one on the left does not have ~ 20k participants.
was referring to "Participants: 19954". I am not sure what the rest of your comment refers to, but I am glad that you see that "The ~20K is the Total Trips (referred from text above the chart)."
I am placing the openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gz file inside em-public-dashboard folder, and executing the following command to load the dataset.
You do not need to place the file in the em-public-dashboard folder. You can specify the fully qualified name instead.
Dropping the Stage_database leads to error in generation of charts "Unable to generate plot Number of trips for each mode (selected by users) Reason:".
What is the reason? You should not have the Stage_database loaded at the same time. I bet the issue is that you are not specifying the alternate DB name openpath_prod_usaid_laos_ev in the DB environment variable. We had discussed this when you first started - @achasmita had this issue and we discussed a resolution back in August.
If you are using a database that is not called Stage_database you need to set the DB environment variable to the appropriate mongo:// URL. Note that you may need to change the start script to use _ instead of -
iantei
commented
8 months ago Similar to @achasmita 's changes -
I incorporated the following changes on docker-compose.yml and start_notebook.sh
Changes in docker-compose.yml
ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73a9875..dc8331c 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -18,10 +18,10 @@ services:
depends_on:
- db
environment:
- - DB_HOST=db
+ - DB_HOST="mongodb://db/openpath_prod_usaid_laos_ev"
- WEB_SERVER_HOST=0.0.0.0
- CRON_MODE=TRUE
- - STUDY_CONFIG=stage-program
+ - STUDY_CONFIG=usaid-laos-ev
ports:
# ipynb in numbers
Changes in start_notebook.sh
ashrest2-35384s:docker ashrest2$ git diff start_notebook.sh
diff --git a/viz_scripts/docker/start_notebook.sh b/viz_scripts/docker/start_notebook.sh
index c02cf29..f40a1ce 100755
--- a/viz_scripts/docker/start_notebook.sh
+++ b/viz_scripts/docker/start_notebook.sh
@@ -7,7 +7,7 @@ if [ -z ${DB_HOST} ] ; then
local_host=`hostname -i`
sed "s_localhost_${local_host}_" conf/storage/db.conf.sample > conf/storage/db.conf
else
- sed "s_localhost_${DB_HOST}_" conf/storage/db.conf.sample > conf/storage/db.conf
+ sed "s-localhost-${DB_HOST}-" conf/storage/db.conf.sample > conf/storage/db.conf
fi
### configure the saved-notebooks directory for persistent notebooks> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.758GBThis yields in json loading error.
``` JSONDecodeError Traceback (most recent call last) Cell In[2], line 7 4 import pandas as pd 6 from plots import * ----> 7 import scaffolding 9 sns.set_style("whitegrid") 10 sns.set() File /usr/src/app/saved-notebooks/scaffolding.py:16 10 # Module for pretty-printing outputs (e.g. head) to help users 11 # understand what is going on 12 # However, this means that this module can only be used in an ipython notebook 14 import IPython.display as disp ---> 16 import emission.core.get_database as edb 18 def no_traceback_handler(exception_type, exception, traceback): 19 print("%s: %s" % (exception_type.__name__, exception), file=sys.stderr) File /usr/src/app/emission/core/get_database.py:19 16 print("storage not configured, falling back to sample, default configuration") 17 config_file = open('conf/storage/db.conf.sample') ---> 19 config_data = json.load(config_file) 20 url = config_data["timeseries"]["url"] 21 result_limit = config_data["timeseries"]["result_limit"] File ~/miniconda-23.1.0/envs/emission/lib/python3.9/json/__init__.py:293, in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 274 def load(fp, *, cls=None, object_hook=None, parse_float=None, 275 parse_int=None, parse_constant=None, object_pairs_hook=None, **kw): 276 """Deserialize ``fp`` (a ``.read()``-supporting file-like object containing 277 a JSON document) to a Python object. 278 (...) 291 kwarg; otherwise ``JSONDecoder`` is used. 292 """ --> 293 return loads(fp.read(), 294 cls=cls, object_hook=object_hook, 295 parse_float=parse_float, parse_int=parse_int, 296 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File ~/miniconda-23.1.0/envs/emission/lib/python3.9/json/__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 341 s = s.decode(detect_encoding(s), 'surrogatepass') 343 if (cls is None and object_hook is None and 344 parse_int is None and parse_float is None and 345 parse_constant is None and object_pairs_hook is None and not kw): --> 346 return _default_decoder.decode(s) 347 if cls is None: 348 cls = JSONDecoder File ~/miniconda-23.1.0/envs/emission/lib/python3.9/json/decoder.py:337, in JSONDecoder.decode(self, s, _w) 332 def decode(self, s, _w=WHITESPACE.match): 333 """Return the Python representation of ``s`` (a ``str`` instance 334 containing a JSON document). 335 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end() 339 if end != len(s): File ~/miniconda-23.1.0/envs/emission/lib/python3.9/json/decoder.py:355, in JSONDecoder.raw_decode(self, s, idx) 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end ```
The dataset is available as seen with the show dbs command, but there's still some issue with it.
iantei
commented
8 months ago There were a few changes which were required in the above approach:
I have to execute docker-compose -f docker-compose.yml build before up command.
NOTE: I will rearrange the snapshots accordingly if needed.
For docker-compose.yml: ``` ashrest2-35384s:em-public-dashboard ashrest2$ git diff docker-compose.yml diff --git a/docker-compose.yml b/docker-compose.yml index 73a9875..8b3a3ba 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -19,9 +19,10 @@ services: - db environment: - DB_HOST=db + - DB_HOST=mongodb://db/openpath_prod_usaid_laos_ev - WEB_SERVER_HOST=0.0.0.0 - CRON_MODE=TRUE - - STUDY_CONFIG=stage-program + - STUDY_CONFIG=usaid-laos-ev ports: # ipynb in numbers - "47962:8888" ``` For start_notebook.sh: ``` - sed "s_localhost_${DB_HOST}_" conf/storage/db.conf.sample > conf/storage/db.conf + sed "s-localhost-${DB_HOST}-" conf/storage/db.conf.sample > conf/storage/db.conf ``` Database changes: ``` > show dbs admin 0.000GB config 0.000GB local 0.000GB > show dbs admin 0.000GB config 0.000GB local 0.000GB openpath_prod_usaid_laos_ev 0.757GB ```
Execution Process:
docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml upMake sure the changes in start_notebook.sh is being reflected inside the docker container.
Execute Jupyter notebooks:
ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-notebook-server-1 /bin/bash
root@fe3d5eb64501:/usr/src/app# source setup/activate.sh
(emission) root@fe3d5eb64501:/usr/src/app# cd saved-notebooks
(emission) root@fe3d5eb64501:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@fe3d5eb64501:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-05T23:10:55.325373+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
(emission) root@fe3d5eb64501:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-05T23:12:47.282348+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (
(emission) root@fe3d5eb64501:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_timeseries.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-05T23:14:23.334011+00:00 with args Namespace(plot_notebook='generic_timeseries.ipynb', program='default', date=None) for range (
Results:
Dataset used: openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gz
Comparison between charts in https://usaid-laos-ev-openpath.nrel.gov/public/ vs http://localhost:3274/?study_config=usaid-laos-ev
| Production | Production container |
|---|---|
These comparison would not be a valid one, since there would be new data every other day. So, comparing between 20th Dec 2023 dataset with 5th Jan 2024 dataset would not yield a rightful comparison.
iantei
commented
8 months ago Comparing between charts generated for both 20th Dec 2023 dataset.
| (https://usaid-laos-ev-openpath.nrel.gov/public/) | (http://localhost:3274/?study_config=usaid-laos-ev) |
|---|---|
Update: The number of confirmed trips and total trips is increased on the one captured in the production site. Upon further detail analysis, All the individual data within the each of the charts has increased values or has remained same. I think the data might have been updated from the point when the data for dataset was collected.
| Comparing individual chart: For Metric : Number of trips All data Default | (https://usaid-laos-ev-openpath.nrel.gov/public/) | (http://localhost:3274/?study_config=usaid-laos-ev) |
|---|---|---|
There seems to be some discrepancy with these two chart.
Possible cause: Each time I load the dataset by deleting the existing database, I get different size of database for the same dataset. Loading the dataset using the following command:
bash viz_scripts/docker/load_mongodump.sh openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gzResult: Different database size for same dataset-
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.756GB
> use openpath_prod_usaid_laos_ev
switched to db openpath_prod_usaid_laos_ev
> db.dropDatabase()
{ "dropped" : "openpath_prod_usaid_laos_ev", "ok" : 1 }
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.757GB
> use openpath_prod_usaid_laos_ev
switched to db openpath_prod_usaid_laos_ev
> db.dropDatabase()
{ "dropped" : "openpath_prod_usaid_laos_ev", "ok" : 1 }
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.755GB
> use openpath_prod_usaid_laos_ev
switched to db openpath_prod_usaid_laos_ev
> db.dropDatabase()
{ "dropped" : "openpath_prod_usaid_laos_ev", "ok" : 1 }
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.757GB
> use openpath_prod_usaid_laos_ev
switched to db openpath_prod_usaid_laos_ev
> db.dropDatabase()
{ "dropped" : "openpath_prod_usaid_laos_ev", "ok" : 1 }
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
> show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.758GBUpdate: In last both cases, where the db size were 0.758GB and 0.757GB - the 4441126 document(s) restored successfully. 0 document(s) failed to restore.
The number of documents restored has been same.
iantei
commented
8 months ago Inside the openpath_prod_usaid_laos_ev database, there are the following collections:
show dbs Stage_database 0.000GB admin 0.000GB config 0.000GB local 0.000GB openpath_prod_usaid_laos_ev 0.767GB use openpath_prod_usaid_laos_ev switched to db openpath_prod_usaid_laos_ev show collections Stage_Profiles Stage_analysis_timeseries Stage_pipeline_state Stage_push_token_mapping Stage_timeseries Stage_timeseries_error Stage_updateable_models Stage_usercache Stage_uuids
Deleted all data which has timestamp greater than 2023-12-20T03:27:00
db.Stage_analysis_timeseries.find({ "data.fmt_time": { $gte: "2023-12-20T03:27:00" } }).count() 0 db.Stage_analysis_timeseries.find({ "metadata.write_fmt_time": { $gte: "2023-12-20T03:27:00" } }).count() 0 db.Stage_timeseries.find({ "metadata.write_fmt_time": { $gte: "2023-12-20T03:27:00" } }).count() 0
Reloaded the Generic Metrics notebook charts.
(emission) root@d839eeb20ee0:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@d839eeb20ee0:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-08T18:22:07.613942+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (
Cleared cache from the Firefox browser. Reloaded the charts.
Result I do not see any difference in the charts. There is still disparity in the confirmed trips and total trips.
iantei
commented
8 months ago @shankari
The results are identical in both snapshot of https://usaid-laos-ev-openpath.nrel.gov/public/ on 20th December 2023 and for the charts generated using production docker compose with the dataset openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gz.
Here we are loading the dataset, and validating the changes are reflecting as the snapshot on the production website.
Steps involved:
Loading the dataset openpath-prod-usaid-laos-ev-snapshot-dec-20.tar.gz for the docker-compose.yml and testing it against the snapshot of https://usaid-laos-ev-openpath.nrel.gov/public/ on 20th December 2023.
After loading the open path_prod_usaid_laos_ev dataset into the database.
> show dbs
Stage_database 0.000GB
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.757GBExecuted the following command for Generic Metrics notebook:
ashrest2-35384s:em-public-dashboard ashrest2$ docker exec -it em-public-dashboard-notebook-server-1 /bin/bash
root@e74c321b6e79:/usr/src/app# source setup/activate.sh
(emission) root@e74c321b6e79:/usr/src/app# cd saved-notebooks
(emission) root@e74c321b6e79:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@e74c321b6e79:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-11T00:07:51.138520+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2023-05-01T00:00:00+00:00]>, <Arrow [2024-01-01T00:00:00+00:00]>)
Running at 2024-01-11T00:07:51.180319+00:00 with params [Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={'MODE': [{'value': 'walk', 'baseMode': 'WALKING', 'met_equivalent': 'WALKING', 'kgCo2PerKm': 0}, {'value': 'e-auto_rickshaw', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.085416859}, {'value': 'auto_rickshaw', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.231943784}, {'value': 'motorcycle', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.113143309}, {'value': 'e-bike', 'baseMode': 'E_BIKE', 'met': {'ALL': {'range': [0, -1], 'mets': 4.9}}, 'kgCo2PerKm': 0.00728}, {'value': 'bike', 'baseMode': 'BICYCLING', 'met_equivalent': 'BICYCLING', 'kgCo2PerKm': 0}, {'value': 'drove_alone', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.22031}, {'value': 'shared_ride', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.11015}, {'value': 'e_car_drove_alone', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.08216}, {'value': 'e_car_shared_ride', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.04108}, {'value': 'taxi', 'baseMode': 'TAXI', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.30741}, {'value': 'bus', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'train', 'baseMode': 'TRAIN', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.12256}, {'value': 'free_shuttle', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'air', 'baseMode': 'AIR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.09975}, {'value': 'not_a_trip', 'baseMode': 'UNKNOWN', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}, {'value': 'other', 'baseMode': 'OTHER', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}], 'PURPOSE': [{'value': 'home'}, {'value': 'work'}, {'value': 'at_work'}, {'value': 'school'}, {'value': 'transit_transfer'}, {'value': 'shopping'}, {'value': 'meal'}, {'value': 'pick_drop_person'}, {'value': 'pick_drop_item'}, {'value': 'personal_med'}, {'value': 'access_recreation'}, {'value': 'exercise'}, {'value': 'entertainment'}, {'value': 'religious'}, {'value': 'other'}], 'translations': {'en': {'walk': 'Walk', 'e-auto_rickshaw': 'E-tuk tuk', 'auto_rickshaw': 'Tuk Tuk', 'motorcycle': 'Motorcycle', 'e-bike': 'E-bike', 'bike': 'Bicycle', 'drove_alone': 'Car Drove Alone', 'shared_ride': 'Car Shared Ride', 'e_car_drove_alone': 'E-Car Drove Alone', 'e_car_shared_ride': 'E-Car Shared Ride', 'taxi': 'Taxi/Loca/inDrive', 'bus': 'Bus', 'train': 'Train', 'free_shuttle': 'Free Shuttle', 'air': 'Airplane', 'not_a_trip': 'Not a trip', 'home': 'Home', 'work': 'To Work', 'at_work': 'At Work', 'school': 'School', 'transit_transfer': 'Transit transfer', 'shopping': 'Shopping', 'meal': 'Meal', 'pick_drop_person': 'Pick-up/ Drop off Person', 'pick_drop_item': 'Pick-up/ Drop off Item', 'personal_med': 'Personal/ Medical', 'access_recreation': 'Access Recreation', 'exercise': 'Recreation/ Exercise', 'entertainment': 'Entertainment/ Social', 'religious': 'Religious', 'other': 'Other'}, 'lo': {'walk': 'ດ້ວຍການຍ່າງ', 'e-auto_rickshaw': 'ລົດ 3 ລໍ້ໄຟຟ້າ ຫລື ຕຸກຕຸກໄຟຟ້າ', 'auto_rickshaw': 'ເດີນທາດ້ວຍ ລົດຕຸກຕຸກ ຫລື ລົດສາມລໍ້', 'motorcycle': 'ລົດຈັກ', 'e-bike': 'ວຍລົດຈັກໄຟຟ້າ', 'bike': 'ລົດຖີບ', 'drove_alone': 'ເດີນທາງ ດ້ວຍລົດໃຫ່ຍ ເຊີ່ງເປັນລົດທີ່ຂັບເອງ', 'shared_ride': 'ເດີນທາງດ້ວຍລົດໃຫ່ຍ ຮ່ວມກັບລົດຄົນອຶ່ນ', 'e_car_drove_alone': 'ດ້ວຍການຂັບລົດໄຟຟ້າໄປເອງ', 'e_car_shared_ride': 'ດ້ວຍການຈ້າງລົດໄຟຟ້າໄປ', 'taxi': 'ແທັກຊີ', 'bus': 'ລົດເມ', 'train': 'ລົດໄຟ', 'free_shuttle': 'ລົດຮັບສົ່ງຟຣີ', 'air': 'ຍົນ', 'not_a_trip': 'ບໍ່ແມ່ນການເດີນທາງ', 'home': 'ບ້ານ', 'work': 'ໄປເຮັດວຽກ', 'at_work': 'ຢູ່ບ່ອນເຮັດວຽກ', 'school': 'ໄປໂຮງຮຽນ', 'transit_transfer': 'ການຖ່າຍໂອນການເດີນທາງ', 'shopping': 'ຊອບປິ້ງ', 'meal': 'ອາຫານ', 'pick_drop_person': 'ໄປຮັບ ຫລື ສົນ ຄົນ', 'pick_drop_item': 'ໄປຮັບ ຫລື ສົ່ງສິນຄ້າ', 'personal_med': 'ໄປຫາໝໍ', 'access_recreation': 'ເຂົ້າເຖິງການພັກຜ່ອນ', 'exercise': 'ພັກຜ່ອນ/ອອກກຳລັງກາຍ', 'entertainment': 'ບັນເທີງ/ສັງຄົມ', 'religious': 'ຈຸດປະສົງທາງສາດສະໜາ', 'other': 'ອື່ນໆ'}}})]Introduced a new function in scaffolding.py
def remove_confirmed_trip_labels():
TS_LIMIT = 1703042805
#first, remove all trips written after a cutoff time
print("counting records for removal")
print(edb.get_timeseries_db().count_documents({ "metadata.write_ts": { "$gt": TS_LIMIT } }))
print(edb.get_analysis_timeseries_db().count_documents({ "metadata.write_ts": { "$gt": TS_LIMIT } }))
#then, find manual inputs added after the cutoff time
print("counting manual records after cutoff")
print(edb.get_timeseries_db().count_documents({"metadata.write_ts": {"$gt": TS_LIMIT}, "metadata.key": {"$regex": '^manual/(mode_confirm|purpose_confirm|replaced_mode)$'}}))
for t in list(edb.get_timeseries_db().find({"metadata.write_ts": {"$gt": TS_LIMIT}, "metadata.key": {"$regex": '^manual/(mode_confirm|purpose_confirm|replaced_mode)$'}})):
confirmed_trip = edb.get_analysis_timeseries_db().find_one({"user_id": t["user_id"],
"metadata.key": "analysis/confirmed_trip",
"data.start_ts": t["data"]["start_ts"]}) #gets confirmed trip with matching user id & timestamp
if confirmed_trip is None:
print("No matching confirmed trip for %s" % t["data"]["start_fmt_time"])
continue
if confirmed_trip["data"]["user_input"] == {}:
print("Found confirmed trip with matching inferred trip, without user labels")
else:
print("Getting here")
update_results = edb.get_analysis_timeseries_db().update_one({"user_id": t["user_id"],
"metadata.key": "analysis/confirmed_trip",
"data.start_ts": t["data"]["start_ts"]}, { "$set": { 'data.user_input': {} } })
print("Update results")
print(update_results)
print("delete after timestamp")
edb.get_timeseries_db().delete_many({ "metadata.write_ts": { "$gt": TS_LIMIT } })
edb.get_analysis_timeseries_db().delete_many({ "metadata.write_ts": { "$gt": TS_LIMIT } })This is called in Generic Metrics notebook, before call to scaffolding.load_viz_notebook_data() as:
scaffolding.remove_confirmed_trip_labels()
Mongo DB after the removal of the entries later than 20th Dec 2023 3:28:45 GMT and removing the extra confirmed trips.
> show dbs
Stage_database 0.000GB
admin 0.000GB
config 0.000GB
local 0.000GB
openpath_prod_usaid_laos_ev 0.771GBResults:
| Overall comparison: | Production snapshot | Docker-compose prod snapshot |
|---|---|---|
| Individual comparison: | Production snapshot | Docker-compose prod snapshot |
|---|---|---|
Thank you @Abby-Wheelis for the suggestions and support in completing this.
shankari
commented
8 months ago For the record, the code used here is not what I suggested
confirmed_trip = edb.get_analysis_timeseries_db().find_one({"user_id": t["user_id"],
"metadata.key": "analysis/confirmed_trip",
"data.start_ts": t["data"]["start_ts"]}) #gets confirmed trip with matching user id & timestampThis is not the matching algorithm. This will not work for labels on draft trips, for example. I had asked you to run the matching algorithm from the server code - e.g. something like
matching_confirmed_trip = ea...find_matching_confirmed_trip(me)@iantei can you please report the results after fixing the code? I anticipate they will be similar, but the goal is is to check the calculation so we want to make the data massaging as accurate as possible.
Abby-Wheelis
commented
8 months ago @shankari is there a specific matching algorithm in the server code that you're thinking of? From ea.. that would indicate that we should focus on emission/analysis, but I'm not sure I've been able to find what you're referring to.
Ok, wait, digging in a little deeper today I've found something that looks promising:
get_confirmed_obj_for_user_input_obj(ts, ui_obj): in trip_queries.py takes a timestamp and a user input object, and looks like it selects the "analysis/confirmed_trip" object that matches.
There is an example of how this function has been used in match_incoming_inputs of matcher.py which seems more focused on an individual user than we need to be, but could be a good example of context.
Maybe that's a good place to start, I had skimmed over it yesterday because it looked like it dealt with place user inputs, but I think it's more general than I first thought.
iantei
commented
8 months ago Inside emission/storage/decorations/trip_queries.py there is a function, get_confirmed_obj_for_user_input_obj(ts, ui_obj): which makes use of potential_candidates = ts.find_entries(["analysis/confirmed_trip"], tq).
This get_confirmed_obj_for_user_input_obj is being used from match_incoming_inputs which is available in emission/analysis/userinput/matcher.py.
Since we are not looking for specific user_id, I tried to use the code available inside get_confirmed_obj_for_user_input_obj.
Use potential_candidates = ts.find_entries(["analysis/confirmed_trip"], tq), to list out the confirmed trips directly.
So, in the function get_confirmed_obj_for_user_input_obj, there is call to final_candidate(valid_timeline_entry(ts, ui_obj), potential_candidates). I am not sure if this is required in our case or not.
[final_candidate() does filter out and return a single item, so I might have to incorporate that too. ]
Even with the potential_candidates, we have a list. So, I am not sure which item from the list to choose from.
for t in list(edb.get_timeseries_db().find({"metadata.write_ts": {"$gt": TS_LIMIT}, "metadata.key": {"$regex": '^manual/(mode_confirm|purpose_confirm|replaced_mode)$'}})):
ts = esta.TimeSeries.get_aggregate_time_series()
ONE_DAY = 24 * 60 * 60
if 'data' in t and 'start_ts' in t['data']:
start_ts_value = t['data']['start_ts']
else:
pass
tq = estt.TimeQuery("data.start_ts", start_ts_value - ONE_DAY,
start_ts_value + ONE_DAY)
# iff the input's key is one of these, the input belongs on a place
# all other keys are only used for trip inputs
place_keys = ["manual/place_user_input", "manual/place_addition_input"]
# print("Hello")
confirmed_trip_list = list(ts.find_entries(["analysis/confirmed_trip"], tq))
if confirmed_trip_list is None:
print("No matching confirmed trip for %s" % t["data"]["start_fmt_time"])
continue
confirmed_trip = confirmed_trip_list[0]
if confirmed_trip and 'data' in confirmed_trip and 'user_input' in confirmed_trip['data'] and not confirmed_trip['data']['user_input']:
# print("Confirmed trip has an empty user_input")
pass
else:
print("Confirmed trip either doesn't exist or has non-empty user_input")
update_results = edb.get_analysis_timeseries_db().update_one({"user_id": t["user_id"],
"metadata.key": "analysis/confirmed_trip",
"data.start_ts": t["data"]["start_ts"]}, { "$set": { 'data.user_input': {} } })
confirmed_trip["data"]["user_input"] = {}@shankari Could you please let know if I looking into the right direction?
Confirmed using https://openpath-stage.nrel.gov/public/?study_config=smart-commute-ebike which does not have the energy impact calculation