Telemetry processing & storage solution

[iiacoban42]

Find a db solution for storing parsed telemetry

SQL vs NoSQL overview:

1. SQL databases are relational, NoSQL databases are non-relational.
2. SQL databases use structured query language and have a predefined schema. NoSQL databases have dynamic schemas for unstructured data.
3. SQL databases are vertically scalable, while NoSQL databases are horizontally scalable.
4. SQL databases are table-based, while NoSQL databases are document, key-value, graph, or wide-column stores.
5. SQL databases are better for multi-row transactions, while NoSQL is better for unstructured data like documents or JSON.

Notes:

• Most features of a relational db are not particularly valuable in our case: no transactions and no concurrency when writing + unstructured data
• Pro for NoSQL: could easily change the schema over time. Actually, they are schemaless so there is no need to worry about changes in the data model.

NoSQL options:

Time series db:

Useful links (data parsing):

• xtcetools wiki
• DelfiC3 xtce
• DelfiPQ transmission

[iiacoban42]

Preliminary choice : MongoDB

• supports time series and integrates with Grafana
• data is JSON which we already use
• can easily convert JSON to pandas to do some ML with the data
• no schema to tell us what's illegal
• it's very popular, hence a lot of community support & solutions to common problems