feat: Add get_progress and set_progress to redis result backend

taskiq-python / taskiq-redis

Broker and result backend for taskiq

MIT License

40 stars 18 forks source link

feat: Add get_progress and set_progress to redis result backend #67

Closed sminnee closed 2 months ago

sminnee commented 4 months ago

Uses as standard suffix on the redis key (hardcoded as "__progress") to store progress results.

sminnee commented 4 months ago

A few questions about this:

Is a key with suffix __progress a good default? Should the suffix be configurable?
When should the data expire? Using redis.getdel() as with get_results is probably not appropriate as it's much more likely to be fetched repeatedly, but maybe it should have a shorter default expiry time? (2h?)
Should it be possible to configure progress expiry separately from results expiry?

I also note that there's a fair bit of duplication in RedisAsyncClusterResultBackend, RedisAsyncResultBackend, and RedisAsyncSentinelResultBackend. They could possibly be refactored to have a common base class with most of the implementation, and the subclass just defining a method that provides the redis connection object as a resource (i.e. with enter() and exit() even if it's not strictly necessary). That refactor is beyond the scope of this PR, but I'm happy to do it.

CC @Sobes76rus as you did the original set_/get_progress implementation

Sobes76rus commented 4 months ago

Hello, I think the __progress suffix is good enough and doesn't need configuration Expiration should be customisable, and it would be even better if it is customisable for each task for results and for progress based on labels

sminnee commented 4 months ago

OK so should I add 2 more arguments to the result backend(s) constructor?

progress_ex_time: Optional[int] = 600
progress_px_time: Optional[int] = None

I'm happy to add that to this PR, if @s3rius gives a 👍.

The bit I'm least confident about is the 600 default - it's a little arbitrary, but I think some default is better than no expiry. My rationale is that if there have been no progress updates in the last 10 minutes, something has probably gone wrong.

sminnee commented 4 months ago

FYI I believe I've fixed the build now, sorry about missing that in the first PR!

sminnee commented 3 months ago

@s3rius it would be great to get your input on this, if you have any? :)

sminnee commented 3 months ago

Hey, just giving this a nudge. I'm running my project on a fork so no great hurry, although it would be good to know if you're on board with the direction I took?

s3rius commented 3 months ago

It looks good to me. Let's check tests and release it if it's fine.

s3rius commented 3 months ago

Can you please rebase onto the main branch? I've fixed the workflows.

sminnee commented 3 months ago

done

pdjohntony commented 2 months ago

Any ETA on this?