search

FleekHQ / brokkr

Background job processing and orchestrator for Node
MIT License
12 stars 3 forks source link

Is redis data kept indefinitely? #5

Open c3s4r opened 4 years ago

c3s4r commented 4 years ago

Hi. I was trying your module for an orchestrator microservice.

From the tests, it looks like even after the saga is complete, the data remains in redis. Is that the intended design?

My use case doesn't require the steps to be persisted after completed. Should I clean the redis DB manually? (I'm worried that I will be storing unnecessary data I don't need anymore).

dmerrill6 commented 4 years ago

Hi @c3s4r!

That's right, currently Brokkr doesn't delete sagas after they are finished in case you want to set up some kind of tracking (for example a dashboard where you can see the status of all pending and finished sagas).

I understand your need to keep data usage low though, specially in the case your orchestrator processes thousands of sagas. I'd love to include a flag so that this could be configured when initializing Brokkr, but don't have the bandwidth right now to do it (would be happy to receive PRs though 😄). Another option is, as you say, cleaning the data manually. You could do this by using the saga id and directly executing Redis DEL on that key.

c3s4r commented 4 years ago

Thank you for your prompt response. Data is not critical for our use case, so I'm thinking of just using the built-in memory client (or Redis without persistence), but I wanted to confirm if it's a good idea... I wouldn't like to find out that it just keep increasing the memory usage.

If I use the built-in memory client, would that be something to worry about?

dmerrill6 commented 4 years ago

In-memory client was built mainly to run tests without Redis dependency, it will definitely work in production, but the downside is that if your node server crashes, you won't be able to pick up the jobs from where it was left off. Also if you run the in-memory client you won't be able to have multiple orchestrator instances.

I think I can make up some time next week to add a Redis clean-up mechanism. I'd recommend you to use the Redis client and wait for the next release if that's possible.

c3s4r commented 4 years ago

No worries. I will use the Redis client. Thanks for sharing the module!