CrayLabs / SmartDashboard

SmartSim Dashboard Package.
BSD 2-Clause "Simplified" License
6 stars 2 forks source link

Migrate Incoming Data Structures from Dicts to Defined Schemas #7

Closed MattToast closed 8 months ago

MattToast commented 11 months ago

Description

Currently SSDashboard formats the data used to populate the dashboard UI into plain-old python dicts of dict[str, Any]. This is functional but is prone to typo errors, is unable to validate incoming data, and is generally just hard to read as a developer. We should consider moving the dict[str, Any] types to unique schemas which are themselves a defined type.

Justification

Parsing the dicts into distinct schemas would make the code more readable, allow us to attach methods to them so that they can handle their own parsing/validating/mutating, and play nicer when preforming static analysis. It will also make the code more portable as well be able to reuse the schema if (and more to the point when) we migrate from dumping data to the file system and move to a more traditional database/event driven architecture.

Implementation Strategy

SSDashboard already has a pretty well defined schema for each of the dict[str, Any] entities. It just needs to be fromalized into proper python schemas/objects/etc. It probably makes most sense for SSDashboard to use an existing library as validating incoming data can be a particularly non trivial task.

Several libraries allow for parsing and validating incoming data to python applications, bhe most widely used is pydantic, which allows users define schemas as python classes. Developers can also attach validators to fields to ensure that data matches the shape/type/etc. that they expect it to be.

These classes then work as regular python classes which allow developers to gain all of the benefits that come from such (better static analysis, better runtime errors, etc.)

Furthermore because pydanitc is so largely adopted in the python community, there are several orm libraries that are able to use pydantic models to build out schemas. This may help aid moving the dashboard away from writing directly to the file system and instead leverage a proper database to persistently store information.