We currently have a default (TypeormDatabase) implementation of the processor store.
For non-API cases having a Postgres as a backend database is an overkill and a more lightweight SQLite may be a better choice:
It's transactional
Lightweight: It doesn't require an additional service to run
The limitation of SQLite is that it supports only a single writer, but that's exactly the setup we have (only the squid processor writes).
SQLite may also be a good choice for API-less pipelines with non-transactional data sinks.
Acceptance criteria:
SQLLite implementation of the Store interface (probably only very minor modifications are needed for TypeormDatabase)
Document the limitations of SQLLite store and intended use-cases
A sample squid implementation with a SQLite store + tests
We currently have a default (TypeormDatabase) implementation of the processor store. For non-API cases having a Postgres as a backend database is an overkill and a more lightweight SQLite may be a better choice:
The limitation of SQLite is that it supports only a single writer, but that's exactly the setup we have (only the squid processor writes). SQLite may also be a good choice for API-less pipelines with non-transactional data sinks.
Acceptance criteria: