search

lanl / dsi

LANL Data Science Infrastructure Project
https://lanl.github.io/dsi
5 stars 4 forks source link

SQLAlchemy DSI backend integration #28

Open jpulidojr opened 1 year ago

jpulidojr commented 1 year ago

Full SQLAlchemy backend integration after plugin/driver refactor is completed

qwofford commented 1 year ago

Should the SQLAlchemy piece be part of the middleware?

jpulidojr commented 1 year ago

SQLAlchemy can be middleware since it has its own abstract language to create databases, but we can also treat is as a driver because SQLAlchemy has access to multiple types of databases for non-relational data light graph databases.

In SQLAlchemy's dictionary, our "Drivers" are called "Dialects", and that toolkit gives you access to PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server.

cmahrens commented 1 year ago

We asked this also and we discussed this at the Monday meeting. I think Quincy was pushing for it to be middleware.

cmahrens commented 1 year ago

Well, I mean "core" if that is what you are also meaning by middleware.

qwofford commented 1 year ago

Daniel is taking a look at Pydantic which might be helpful in conjunction with SQLAlchemy. See #51 and https://github.com/tiangolo/pydantic-sqlalchemy

qwofford commented 1 year ago

Optional Pydantic type checking is now in the main branch. If a Plugin implements type checking it can be static or dynamic. An example of static Plugin enforcement can be seen in the Hostname plugin validation model. An example of the dynamic plugin can be seen in the Bueno plugin validation model. It's still possible to create Plugins without Pydantic type enforcement if desired.

Daniel and I are working on code re-structure and documentation based on feedback we received from the group last week, and you can expect changes soon.