danielgerlag / workflow-core

Lightweight workflow engine for .NET Standard
MIT License
5.4k stars 1.2k forks source link

parallel task works sequentially #1223

Closed JeffrinGirin closed 2 weeks ago

JeffrinGirin commented 11 months ago

Hi, I need to excute three methods in parallel but the following code excute it in sequential order. Kindly help me to sort the issue.

Also, tried WaitFor & Delay they didn't work as expected. They are waiting for the first branch to finish.

public class WorkFlow_BusinessService : IWorkflow<InputData>
{
    public string Id => "Step1";
    public int Version => 1;
    public void Build(IWorkflowBuilder<InputData> builder)
    {
        builder
            .Parallel()
                .Do(then =>
                    then.StartWith<Data1>()
                            .Input(step => step.inputObj, data => data)
                            .Output((step, data) => step.OutputObj))
                .Do(then =>
                    then.StartWith<Data2>()
                            .Input(step => step.inputObj, data => data)
                            .Output((step, data) => step.OutputObj)
                        .Then<Data2A>()
                            .Input(step => step.inputObj, data => data)
                            .Output((step, data) => step.OutputObj))
                .Do(then =>
                    then.StartWith<Data3>()
                            .Input(step => step.inputObj, data => data)
                            .Output((step, data) => step.OutputObj))
            .Join();
    }

}

public class Data1 : StepBody
{
    public InputData inputObj { get; set; }
    public InputData OutputObj { get; set; }

    public override ExecutionResult Run(IStepExecutionContext context)
    {
        Console.WriteLine("Data1 Started at " + DateTime.Now.ToString());

        OutputObj = // Internal Method

        Console.WriteLine("Data1 Completed at " + DateTime.Now.ToString());

        return ExecutionResult.Next();
    }

}

Output Data1 started at 1/3/2024 15:09:28 Data1 Completed at 1/3/2024 15:09:29

Data2 started at 1/3/2024 15:09:29 Data2 Completed at 1/3/2024 15:09:31

Data2A started at 1/3/2024 15:09:34 Data2A Completed at 1/3/2024 15:09:37

Data3 started at 1/3/2024 15:09:31 Data3 Completed at 1/3/2024 15:09:34

Expected Output

Data1 started at 1/3/2024 15:09:28 Data1 Completed at 1/3/2024 15:09:29

Data2 started at 1/3/2024 15:09:28 Data2 Completed at 1/3/2024 15:09:30

Data2A started at 1/3/2024 15:09:30 Data2A Completed at 1/3/2024 15:09:33

Data3 started at 1/3/2024 15:09:28 Data3 Completed at 1/3/2024 15:09:31

cjundt commented 10 months ago

Hi, For long running steps you should consider using thing like :

var taskId = context.PersistenceData;
if(taskId==null){
    taskId = StartLongRunningTask();
    return ExecutionResult.Sleep( 500, taskId);
}
var status = GetLongRunningTaskStatus(taskId);
switch(status) {
    case "OK":
        return ExecutionResult.Next( );
    case "Running":
        return ExecutionResult.Sleep( 500, taskId);
}

because next step won't be launched until current step Run method returns.

AngrySKL commented 9 months ago

I am facing the same issue here, even for the Sample09 the soure code provides works in sequence, have u solved the problem yet?

sunnyzees commented 2 weeks ago

I am facing similar issue. how do we run steps in parallel?

danielgerlag commented 2 weeks ago

That is by design. Since workflows are long running background processes, parallel in this context means that each parallel branch will not block another branch. So if you have a WaitFor or a Delay, etc... in one of the parallel branches, the others will still be free to execute.

sunnyzees commented 2 weeks ago

I tried to put delay as well but I doesn't really execute in parallel. I have a workflow where I am getting data from three different APIs which I want to run in parallel and final activity which waits for previous three activities. Can it be achieved ?

On Fri, Nov 15, 2024, 12:42 PM Daniel Gerlag @.***> wrote:

That is by design. Since workflows are long running background processes, parallel in this context means that each parallel branch will not block another branch. So if you have a WaitFor or a Delay, etc... in one of the parallel branches, the others will still be free to execute.

— Reply to this email directly, view it on GitHub https://github.com/danielgerlag/workflow-core/issues/1223#issuecomment-2479569823, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTWDXQVUZH3EX24DJKQUST2AYW67AVCNFSM6AAAAABR3NI6IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZZGU3DSOBSGM . You are receiving this because you commented.Message ID: @.***>

danielgerlag commented 2 weeks ago

This is a library for long running workflows. Only one step can be executing code at a time but if the step surrenders control back to the workflow before it is finished, other steps can move forward before the current step is resumed.

danielgerlag commented 2 weeks ago

How did you put the delay in?

sunnyzees commented 2 weeks ago

Ah, so I think that explains. There is no way to achieve true parallelism in this library. To make sure I understand correctly, only step for a particular worklow instance can be running. Also then delay does rnot eally help if we want 3 steps to execute simulatenously

On Fri, Nov 15, 2024, 4:53 PM Daniel Gerlag @.***> wrote:

This is a library for long running workflows. Only one step can be executing code at a time but if the step surrenders control back to the workflow before it is finished, other steps can move forward before the current step is resumed.

— Reply to this email directly, view it on GitHub https://github.com/danielgerlag/workflow-core/issues/1223#issuecomment-2479994353, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTWDXXLQO6PXR5OX53HDCL2AZUMJAVCNFSM6AAAAABR3NI6IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZZHE4TIMZVGM . You are receiving this because you commented.Message ID: @.***>

danielgerlag commented 2 weeks ago

It is while one step is waiting for some external thing, that other steps can move forward. The WaitFor, Activity and Delay steps, have this functionality, you could also implement it in a custom step, with specific ExecutionResult values returned by the step handler, which instructs the workflow engine to come back to this step later, or when some event has occured.

danielgerlag commented 2 weeks ago

It sounds like your use case needs TPL rather than a workflow engine.

sunnyzees commented 1 week ago

Yea but in TPL error recovery would mean running whole operation of getting data from 3 Apis again.

On Fri, Nov 15, 2024, 5:33 PM Daniel Gerlag @.***> wrote:

It sounds like your use case needs TPL rather than a workflow engine.

— Reply to this email directly, view it on GitHub https://github.com/danielgerlag/workflow-core/issues/1223#issuecomment-2480055185, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTWDXV5GBXH3VY5VJSXZSD2AZZDJAVCNFSM6AAAAABR3NI6IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBQGA2TKMJYGU . You are receiving this because you commented.Message ID: @.***>

danielgerlag commented 1 week ago

I think you need something like this https://www.pollydocs.org

sunnyzees commented 1 week ago

I think I need workflow library with state tracking ,persistence, recovery and where whole worlflow is not per thread. .

On Sat, Nov 16, 2024, 12:23 AM Daniel Gerlag @.***> wrote:

I think you need something like this https://www.pollydocs.org

— Reply to this email directly, view it on GitHub https://github.com/danielgerlag/workflow-core/issues/1223#issuecomment-2480412864, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADTWDXSQQP2TYJMR63BLOD32A3JGLAVCNFSM6AAAAABR3NI6IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBQGQYTEOBWGQ . You are receiving this because you commented.Message ID: @.***>