Architectural Proposal: Dashboard Data Handling

[MaanasArora]

Motivation

The current architecture of the dashboard application can benefit from structural and design improvements. Key areas for improvement include:

• Separation of concerns: We do not have a clear separation between presentation, logic, and data. For example, most changes to the metrics we display require changes to the loading architecture.
• Repetition: There is quite a bit of repeated code, especially the SQL queries, the construction of metrics, and the classes for data flow.
• Added complexity: Some parts of the system are redundant or removable.

Proposal

The proposed architecture will consist of three (largely) independent software components: data loading, preprocessing, and display.

Data Loading

This component will handle the creation and execution of queries to WebCOMAND. It can consist of two subcomponents (files?), the database and loading subcomponents. We should probably use the SQLAlchemy library here.

Database

Here, we handle making connections to and sending queries to WebCOMAND in a generalized way. This is the only part of the app where WebCOMAND is accessible.

Classes:

• NoiseDatabase class will handle the WebCOMAND connection.
  • On __init__, create the connection based on the environment.
  • The public async function execute to send queries and return the response (could be a SQLAlchemy object or a NoiseQuery object, see below).

Loading

This component will handle the creation of queries to be sent to NoiseDatabase.

Classes:

• NoiseQuery class to model queries.
• Parameters:
  • level: either "system" or "device"
  • resolution: integer in minutes (?)
  • features: probably a list for now, containing any, some, or all of "average", "min", "max"
  • device (optional) to filter by device
  • start_time (optional) to filter by start time
  • end_time (optional) to filter by end time
• compile to compile to final query for NoiseDatabase.

Preprocessing

We handle any Pandas preprocessing here.

Simple functions with, at a minimum, the argument db: NoiseDatabase should be OK, and we can construct NoiseQuerys inside the functions. E.g. get_outliers(db, weeks=2).

Display

TODO: I am not very familiar with Dash, so I will leave this to @danieltsoukup or discuss further. I am somewhat partial to using a JavaScript or even HTML frontend (rendering with a Python web server or not), however, if we do use Dash, we should probably decouple our implementation a lot further.

But I do note that it is a good idea to abstract out the setup for the application (including setting up NoiseDatabase) as much as possible. So probably two components, Application (for setup) and Display (for the actual view).

Thank you! Hopefully this isn't too jumbled, lol.

[danieltsoukup]

Thanks Maanas for putting this together, I'm keen to improve the design and really appreciate the thought you put into this!

Database

Completely agree that this should be further decoupled from Webcommand 👍 Lets create the NoiseDatabase & NoiseQuery interface but in a way that we can plug in different data sources i.e. a csv file or SQLite db for local testing, Webcommand or other alternative DBs. I'd love to create a design that is easy to adapt to changes to the DB connection or if another city would like to run a similar noise project.

Mitch is currently working on redesigning the Webcommand API so we won't need to send SQL queries any more, the plan is to have REST URL requests in a similar fashion as your Loading interface.

Loading

I guess this would be the level adaptable to the data source? This was my intention originally with separating in src/data_loading.py the AbstractDataLoader class and URLBuilder but it got a bit messy.

Question: in your setup, the NoiseDatabase.execute() method, what parameters does it use? The same as NoiseQuery?

Preprocessing

Definitely approve using pandas over more elaborate queries 👍 Given the flexible query interface, I think this processing can be naturally done as part of the plot creation. Dash allows us to store data on the client side once the query was run so we can easily re-use the same query output for separate plots.

Display

One benefit of Dash is that you only need to know python to set up everything from back-end hosting/serving to front-end layout & design, plus nicely integrated with plotly so I'm definitely for keeping this. We could definitely walk through the app initialization and see how that can be improved but there are some limitations to what needs to be done directly in the app.py vs what we can hide/re-factor in other modules. The goal of creating the components.py module was to move the component and callback creation to the side while app.py can focus on high level functions: load the data -> initialize the components & callbacks -> set layout -> run the app. Would love to hear your thoughts on this and alternatives you have in mind.

FYI take a look at this app and their source, maybe we can learn a few things on how to better structure things https://clima.cbe.berkeley.edu/

[danieltsoukup]

This issue is currently pending on the WebCommand API redesign, once the latter is finalized we will tackle this next.