Describe the bug
When running inngest start the memory usage is quickly increasing over time.
I have a lot of events with a few KBs payload. After a few hours the inngest process is consuming more than 5GB of memory.
The problem happens when using the in-memory persistence and also with an external redis store.
To Reproduce
Add a function that sends a few KBs of data between steps.
Run functions very often
Expected behavior
I would expect that all the data is either in the sqlite database or the redis cache and the memory consumption of the inngest process is low (or not growing infinitely). Only the data for the current runs should be in memory the rest should be persistent somewhere else.
Additional notes
I am okay with restarting the process regularly to clean the memory, but after a restart inngest is crashing (index not found).
At the moment I am erasing everything on restart (sqlite, redis) to get it running again.
linear[bot]
commented
1 hour ago
Describe the bug When running
inngest startthe memory usage is quickly increasing over time. I have a lot of events with a few KBs payload. After a few hours the inngest process is consuming more than 5GB of memory. The problem happens when using the in-memory persistence and also with an external redis store.To Reproduce
Expected behavior I would expect that all the data is either in the sqlite database or the redis cache and the memory consumption of the inngest process is low (or not growing infinitely). Only the data for the current runs should be in memory the rest should be persistent somewhere else.
System info
Additional notes I am okay with restarting the process regularly to clean the memory, but after a restart inngest is crashing (index not found). At the moment I am erasing everything on restart (sqlite, redis) to get it running again.