search

SNEWS2 / SNEWS_Coincidence_System

Coincidence System backend for snews alert trigger
BSD 2-Clause "Simplified" License
1 stars 2 forks source link

What would happen in a World without Mongo? #71

Closed habig closed 3 months ago

habig commented 1 year ago

Mongo started out life as the data structure in which we stored alerts to form a coincidence. Now we do that in with python data structures.

We log heartbeats to csv.

We log alerts to mongo, but we reset Mongo each day due to our data retention idea of "won't keep it fot more than a day".

What if we also logged alerts to a flat file? We already have the machinery to purge old data from the heartbeats. Could re-use that for alerts too.

Not having Mongo eliminates a lot of complexity: don't have to install that, don't have to include it and its ports in the containers.

justinvasel commented 1 year ago

One option would be to replace Mongo with a simple SQLite database. It has all the greatest hits from your favorite SQL databases without the overhead of running a separate process/server. The DB itself can be stored either in memory, or as a file on disk if persistency following a crash/restart is needed.

An Object-Relational Mapper (ORM) wrapper library makes it easy to abstract out using raw SQL queries. SQLModel (pip3 install sqlmodel) is a great one that links the python data model classes to their corresponding DB table schema (and vice-versa). It also handle things like validation, SQL statement sanitation, ensuring transactions are ACID-compliant, etc under the hood.

SQLModel: https://sqlmodel.tiangolo.com/ Simple example of creating a data class and inserting instances of it into a DB: image

KaraMelih commented 3 months ago

I think no longer applies. sqlalchemy is implemented in #106 closing the issue