search

temporalio / temporal

Temporal service
https://docs.temporal.io
MIT License
11.63k stars 829 forks source link

Support sqlite in production. #3366

Open shanna opened 2 years ago

shanna commented 2 years ago

Is your feature request related to a problem? Please describe. I'm a fan of self hosting early stage and startup projects on micro budgets (think 5USD to 100USD a month at most). I've been using Temporal on work projects but more recently I've been playing with Temporalite using sqlite replicated with https://litestream.io to https://min.io. It has worked out great in development but I'd like to take Temporal (or Temporalite) to production without clusting, postgres, elasticsearch etc which all blow out small budgets in a heartbeat.

Describe the solution you'd like Officially support sqlite in production.

I think this would involve (at least):

Describe alternatives you've considered I have been tempted to use temporalite in production despite all the risks and downsides.

Additional context I don't fully utilise Temporal. Generally my workflows are for service to service coordination and short lived so I'm OK with a single nodes and short stretches of downtime which I understand probably isn't typical for Temporal users.

I asked about future plans for sqlite support here https://community.temporal.io/t/future-sqlite3-support/5915?u=shane

jlegrone commented 2 years ago

FYI, the vision is for Temporalite to be used in production contexts. I might make a milestone to make tracking easier, but this depends on https://github.com/temporalio/temporalite/issues/23, https://github.com/temporalio/temporalite/issues/110, and signoff from more Temporal server maintainers. I'd love to get published benchmarks, which are in the works.

Of course you are free to use the regular Temporal server binary with the sqlite driver as well! Any improvements to performance and stability will also be available in Temporalite.

Also, 👍 for litestream. I haven't gotten around to playing with it yet, but glad to know it's working for you with Temporalite!

evgenii-moriakhin commented 7 months ago

Hello

There is no progress or discussion on this issue? I see the problem has been reopened

I'm interested because I too want to use temporal with sqlite on a virtualization product where thousands of hosts may have agents that don't have broad access to the enterprise network, but I'd like to be able to execute temporal workflows on them without having to deploy a "serious" DBMS like PostgreSQL on each host, but take advantage of temporal.

I read this topic https://community.temporal.io/t/future-sqlite3-support/5915/6?u=shane and find the use cases completely similar to mine

and I had the question after reading this topic, if these points

One reason is it starts all server roles in the same container and in a single process (similar to auto-setup) which makes it hard to scale and tune among other things

are not critical for me, can I use temporal with sqlite in production?

@shanna , did you finally get it to work?