shankari
commented
2 years ago This is super simple because I wanted to allow analysis experts to write new metrics using jupyter notebooks and R and whatever they want and then to also not allow very fine-grained data manipulation to avoid privacy leakage through repeated queries.
There is an summer intern who is working on revamping the images to show incorporate analysed results along with error bars and she may be using plotly which seems like the standard in the dataviz team, and also supports R and python. In that case, we may adapt this to be a plotly dashboard as well.
But for now, let's get this in production.
The main challenge with getting this into production is the storage of the generated images. Right now, they are stored in files on a shared volume between the two containers. But cloud services does not like file storage because they want all their containers to be serverless.
So they will want to map this to:
- a simple webapp container
- a serverless schedule task in AWS that runs once a day
So we will have to store the images in some other shared storage. Options are:
- database, OR
- store in in S3
Next steps are (in parallel):
- get the public dashboard to run on your laptop so you know how it works
- figure out path going forward for the public dashboard
- explore plotly
- check with the SULI intern on whether they have finalized on plotly
- check whether this is data viz team has standardized on
- see whether we should switch to plotly as well (explore it a bit, is it easy to use for a transportation analyst, is it likely to be well maintained...)
- in that case, do we even need to generate static images, or will we pull from the database to generate on the fly?
- figure out how to store static images
- talk to cloud services about S3 vs. database
- make the appropriate changes
- work with IT to deploy the webapp and any scheduled task into the cloud service per enviroment
- this will also involve adding a route to our custom ngnix https://github.nrel.gov/nrel-cloud-computing/openpath-nginx to the new webapp (e.g. https://
-openpath.nrel.gov/public_dashboard)
- explore plotly
The public dashboard is currently at: https://github.com/e-mission/em-public-dashboard
It is a static dashboard - it creates images (updated daily) for the aggregate metrics for the current month It has a very simple frontend that allows people to choose the month and program and then displays the static image
It essentially provisions two docker containers: one for generating the static images (that we basically run notebooks using a cronjob to generate) and a dead simple frontend using jquery that basically displays the image.