Implement persistance

[mvSapphire]

Low priority

It might be helpful to have some persistence for pipelines. On error, all pipeline and context should be serialized and stored, for example in Redis (IMO Redis should be implemented first). On the next execution pipeline state should be restored from the DB and the pipeline should proceed where it stopped

[caesay]

Just to add a suggestion - job/pipeline persistence is often handled with some kind of message bus (eg. Kafka, Azure Service Bus). A message can sit in a queue until it's handled, if there's an error it can be returned to the queue to try again later. Critically this separates the "resilience" responsibility from your main code, and allows you to build a single workflow which span multiple applications (eg., you have a separate "EmailService" which is responsible for handling the CustomerNotificationsStep from your example). Is a message bus backend something you'd consider?

[homolibere]

It's an interesting idea. This is just a concept we want to think about. Thanks for your interest in this.

As for message bus or something like this, we have steps that are not controlled by us and they can contain several operations in them. Those operations can be not idempotent, for example inserting records in DB. If we restore message for this step - it will go from beginning and fails on DB insert.

@caesay may be you have some solution for this?

This is might work when you know what you do. If during pipeline development you'd specifically code all steps with idempotency in mind and keep them as simple as it can be - this could work. But even in this case all we need to do is to save pipeline context (on step enter) and step where we stopped - restoring only this 2 thing will continue pipeline execution as like interruption never happened.

So, do we need to constrain somehow users for such steps (as single responsibility and idempotent) when they use persistence, or they should control this on their own?

[mvSapphire]

@caesay Thanks for your comment. Message bus could be a good solution. However, this approach requires a lot of architecture changes.

@homolibere I think even if we want to constrain users for such steps I don't see how. However, I think you already answered your question - they should control this on their own 🙂