maxcountryman / underway

⏳ Durable step functions via Postgres.
Apache License 2.0
71 stars 2 forks source link

provide task_attempt table #50

Closed maxcountryman closed 2 weeks ago

maxcountryman commented 2 weeks ago

This reworks how task executions are tracked by providing a new table, task_attempt which is a log of each dequeue of a given task from the queue. Put another way, each time a task is executed, a row is inserted into the attempts table and the result of that execution is stored on that row.

As such, the task table is now a container for the fixed metadata of a specific enqueue of a task. This means that metadata associated with an attempt now lives on the task_attempt table. For example, error messages are stored on the attempts table and likewise creation and completion timestamps also live on this table.

Importantly some fields of the task row have been removed as they are now derived from the attempts log. For instance, task_attempt provides an updated_at column which allows for e.g. determining succeeded_at or failed_at timestamps of the task enqueue. Similarly retry_count is a function of the number of attempt rows associated to a given task.

Furthermore, the retry policy is now embedded as a JSONB column on the task row.

Related discussion: https://github.com/maxcountryman/underway/discussions/17

maxcountryman commented 2 weeks ago

I should point out that I have not added tests here that specifically target task attempt rows. If we agree this is the right direction, I will plan to add those before we release this.

maxcountryman commented 2 weeks ago

This is ready for review.

Testing could still be improved but I've added cases for the state transitions that check attempt rows are being created as we'd expect.

kirillsalykin commented 2 weeks ago

it looks good to me

kirillsalykin commented 2 weeks ago

Maybe I'll have more comments once I start using it

maxcountryman commented 2 weeks ago

@kirillsalykin thanks for all your feedback thus far: very much appreciate it. 🙏