July Cleanups

[Abby-Wheelis]

Starting a PR to address the comments on #136

Planning to hold this for additional cleanups that may be discovered while changes are on staging!

[Abby-Wheelis]

Things that are broken:

• "Timeseries of Total and Labeled Trip Counts" -- all staging
• Number of trips, etc on dfc-fermata (the survey responses look good though!)

[Abby-Wheelis]

errors found in logs:

sensed_algo_prefix is not defined while running the generic_timeseries notebook --> missing parameter definition!!

generate_metrics: error: the following arguments are required: program in survey_metrics notebook --> cause of missing charts on dfc-fermata

[Abby-Wheelis]

The only success I have had with reproducing the generate_metrics: error: the following arguments are required: program error was when I ran the command PYTHONPATH=.. python bin/generate_plots.py survey_metrics.ipynb instead of PYTHONPATH=.. python bin/generate_plots.py survey_metrics.ipynb default on the command line

But in the crontab file the correct command is present, so that is not the issue

[shankari]

generate_metrics: error: the following arguments are required: program in survey_metrics notebook --> cause of missing charts on dfc-fermata

This seems unlikely to be the problem since the survey_metrics notebook generates the survey results and they seem fine. Are there any errors with the generic_metrics notebook?

[shankari]

Ah, I see! The survey_responses notebook is the one that generates the survey graphs. The default values are generated using survey_metrics. In our internal system that runs on AWS, we don't use the crontab, but AWS scheduled tasks, and the generate_plots is in fact missing the program

python bin/generate_plots.py energy_calculations.ipynb default
python bin/generate_plots.py survey_responses.ipynb default
python bin/generate_plots.py survey_metrics.ipynb

Fixing it internally and tagging Abby

[Abby-Wheelis]

I've identified another bug that should be fixed here - every one of the tables in the text display of the survey responses is the same as the last one - I'm guessing I managed to overwrite the whole list every time instead of the correct position

[Abby-Wheelis]

dfc-fermata now working as expected

I do wonder though, why are we showing the sensed modes and not the detected modes? I'm going to look into if we're ready to switch over to the detected modes ...

[Abby-Wheelis]

I do wonder though, why are we showing the sensed modes and not the detected modes? I'm going to look into if we're ready to switch over to the detected modes ...

We're missing the 'ble_sensed_summary' in the test data that I have loaded, at least, is this mainstreamed in the data model yet?

It's in the code base! In server, admin dashboard, and the public dashboard

I at one point had this commented out, but have restored the code ... I can't find an example of it working yet though?

Is it just not in the data sample I have loaded? Dumping and reloading ...

[Abby-Wheelis]

After reloading the May 8th snapshot - bluetooth sensed modes are populating as expected -- most of the data at this point was collected before we were detecting the specific vehicle with bluetooth, so the large % of unknown makes sense.

[Abby-Wheelis]

Also updated the generic_metrics_sensed notebook to use the ble mode

ca-ebike does not use the ble mode:

dfc-fermata does use the ble mode:

I believe those are all the changes needed I will more completely this afternoon!

[Abby-Wheelis]

Tested on dfc-fermata - looks good charts all generate alt text is no longer overly duplicative

Tested on ca-ebike -- labeled trips are broken?

[Abby-Wheelis]

Tested on ca-ebike -- labeled trips are broken?

Totally fine in the notebook though ... clearing everything out and trying again

Just needed to reset my cached images!

[shankari]

@Abby-Wheelis just checking on whether this is done; I would like to merge and push to staging this weekend so that we can unblock the other changes

[Abby-Wheelis]

Yes, this should be ok to go to staging, sorry!