search

cgillum / durabletask-dapr

Reference implementation for the Dapr Workflow embedded engine
https://github.com/dapr/dapr/issues/4576
MIT License
15 stars 0 forks source link

DurableTask-sidecar #17

Closed jkears closed 2 years ago

jkears commented 2 years ago

Hi @cgillum

I am playing with the DurableTask.Dapr POC, and attempting to integrate with some existing test services that I spin up using Tye. With Tye, it does not register HTTP-Port 3500, it's randomized.

However, your durabletask-sidecar image is expecting to communicate at 3500, which is failing when I attempt to invoke a new instance of my test orchestration as follows (Docker DurableTask.sidecar:03.3..) ; 

2022-09-09T04:46:14.425299Z info: Microsoft.DurableTask.Sidecar[1] Initializing the Durable Task sidecar. Listen address = http://0.0.0.0:4001, backend type = Dapr.
2022-09-09T04:46:14.593722Z info: Microsoft.DurableTask.Sidecar[2] Sidecar initialized successfully in 246ms.
2022-09-09T04:48:52.023040Z info: Microsoft.DurableTask.Sidecar[6] Received work-item connection from ipv4:172.17.0.1:34012. Client connection deadline = 9999-12-31T23:59:59.
2022-09-09T04:48:52.029980Z info: Microsoft.Hosting.Lifetime[14] Now listening on: http://[::]:80
2022-09-09T04:48:52.031585Z info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
2022-09-09T04:48:52.032581Z info: Microsoft.Hosting.Lifetime[0] Hosting environment: Production
2022-09-09T04:48:52.032671Z info: Microsoft.Hosting.Lifetime[0] Content root path: /app/
2022-09-09T04:49:54.121175Z info: Microsoft.DurableTask.Sidecar[7] Client at ipv4:172.17.0.1:34012 has disconnected. No further work-items will be processed until a new connection is established.
2022-09-09T04:50:54.121355Z info: Microsoft.DurableTask.Sidecar[5] Waiting for a remote client to connect to this server. Total wait time: 00:00:59.9979291
2022-09-09T04:51:54.122519Z info: Microsoft.DurableTask.Sidecar[5] Waiting for a remote client to connect to this server. Total wait time: 00:02:00.0005585
2022-09-09T04:52:54.123415Z info: Microsoft.DurableTask.Sidecar[5] Waiting for a remote client to connect to this server. Total wait time: 00:03:00.0014612
2022-09-09T04:53:15.365378Z info: Microsoft.DurableTask.Sidecar[6] Received work-item connection from ipv4:172.17.0.1:34140. Client connection deadline = 9999-12-31T23:59:59.
2022-09-09T04:55:00.545869Z info: Microsoft.DurableTask.Sidecar[0] Creating a new instance with ID = 3fa85f64-5717-4562-b3fc-2c963f66afa6
2022-09-09T04:55:00.640902Z fail: Microsoft.DurableTask.Sidecar[0] An error occurred when trying to create a new instance System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:3500)  ---> System.Net.Sockets.SocketException (111): Connection refused    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)    --- End of inner exception stack trace ---    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)    at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleSecurityExceptions(Func`1 requestFunc, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsync(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken)    at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken)    at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)    at DurableTask.Dapr.DaprOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses)    at Microsoft.DurableTask.Sidecar.Grpc.TaskHubGrpcServer.StartInstance(CreateInstanceRequest request, ServerCallContext context) in /root/src/DurableTask.Sidecar/Grpc/TaskHubGrpcServer.cs:line 136
2022-09-09T04:55:00.656905Z fail: Grpc.AspNetCore.Server.ServerCallHandler[6] Error when executing service method 'StartInstance'. System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:3500)  ---> System.Net.Sockets.SocketException (111): Connection refused    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)    at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)    --- End of inner exception stack trace ---    at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)    at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)    at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleSecurityExceptions(Func`1 requestFunc, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsyncHandleUnsuccessfulResponse(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.SendAsync(Func`1 requestFunc, String relativeUri, CancellationToken cancellationToken)    at Dapr.Actors.DaprHttpInteractor.InvokeActorMethodWithRemotingAsync(ActorMessageSerializersManager serializersManager, IActorRequestMessage remotingRequestRequestMessage, CancellationToken cancellationToken)    at Dapr.Actors.Communication.Client.ActorRemotingClient.InvokeAsync(IActorRequestMessage remotingRequestMessage, CancellationToken cancellationToken)    at Dapr.Actors.Client.ActorProxy.InvokeMethodAsync(Int32 interfaceId, Int32 methodId, String methodName, IActorRequestMessageBody requestMsgBodyValue, CancellationToken cancellationToken)    at DurableTask.Dapr.DaprOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses)    at Microsoft.DurableTask.Sidecar.Grpc.TaskHubGrpcServer.StartInstance(CreateInstanceRequest request, ServerCallContext context) in /root/src/DurableTask.Sidecar/Grpc/TaskHubGrpcServer.cs:line 136    at Grpc.Shared.Server.UnaryServerMethodInvoker`3.AwaitInvoker(Task`1 invokerTask, GrpcActivatorHandle`1 serviceHandle)    at Grpc.Shared.Server.UnaryServerMethodInvoker`3.AwaitInvoker(Task`1 invokerTask, GrpcActivatorHandle`1 serviceHandle)    at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)    at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

My hope is to build out and test orchestrations (using Tye) within my current dev environment, such that I can flip them over to your final DAPR Workflow component solution, once it becomes available. I do not intend to push your POC into a production env, but assuming the future DAPR Workflow .Net SDK is similar, then any Orchestrations would/should work with the final DAPR component.

I am new to DurableTask, so was hoping you could please provide me with code that you used to create your POC Docker image so I can run the DurableTask-sidecar dapr service as a Tye invocation, to test my WF integration(s).

Thanks in advance! John

cgillum commented 2 years ago

Hi @jkears. The sidecar image internally uses the Dapr SDK for .NET. I haven't tried this myself, but I think you may be able to set a DAPR_HTTP_PORT environment variable on this container to get it to use a port number other than 3500 (this code doesn't hardcode the port number anywhere).

I am new to DurableTask, so was hoping you could please provide me with code that you used to create your POC Docker image so I can run the DurableTask-sidecar dapr service as a Tye invocation, to test my WF integration(s).

Just to make sure I understand, you're asking if you can get access to the Durable Task sidecar source code so you can build it yourself? I haven't used Tye before, but does it require all containers to be built from source, vs. referencing existing images from dockerhub? Or were you thinking you might need to make changes to the code? Just trying to understand.

jkears commented 2 years ago

Hi @cgillum

Just to make sure I understand, you're asking if you can get access to the Durable Task sidecar source code so you can build it yourself? I haven't used Tye before, but does it require all containers to be built from source, vs. referencing existing images from dockerhub? Or were you thinking you might need to make changes to the code? Just trying to understand.

Yes, and yes to both of your questions;

We developed in-house tooling which provides us an ability to capture any business domain and from which, code-generate (using Roslyn), DAPR based microservices. Our tooling will assist us in creating some future online products that consist of many microservices.

We created a VS Code Web View Extension (see image below) that allows us to define microservices that are code-generated from a DSL based model. We follow an Aggregate-First, Domain Driven Design approach in defining our services.

image

The above is a sample model from our ERP, specifically I am showing the FinTech Modules, General Ledger Service which consists of a bunch of Aggregates (showing JournalConfiguration and JournalBatch for example).

Our tooling allows us to define services consisting of as many or as few aggregates as would be necessary to describe the each business domain, and with great ease, code-generate the Web-API microservices that run within DAPR.

In development, MS Tye allows us to run all of these generated services all with the same DAPR network. With Tye we define the various services in the Tye.yml file (which is also code-generated).

Below is the generated Tye file for the above model...

image

To spin up all code-generated services, we need only run Tye Run...

image

And with that we can attach to any service to debug code, etc.

Tye also provides a dashboard that allows us to view the logs of each microservice as well as it's DAPR side-car..

image

To extend this, and in prep for your soon-to-come, DAPR workflow component, we have created a supporting Aggregate type that is of type Workflow.

image

This will code-generate the above commands to create, integrate, (not shown as not in POC pause/resume) and terminate commands. But instead of persisting to MSSQL via EFCore or CosmosDB, we would use your DurableTask Dapr WF context to create, new instance, terminate, etc...

So ideally, we would like to take what you are doing within the durabletask-sidecar image, and place that into one of our code generated services so that, we can run that service within Tye, as well as learn how it integrates with DAPR SDK .Net.

Once you have released the DT Workflow for DAPR, we would then flip over to use that, and everything should work the exact same but as workflow-actors hosted within each service's side-car.

This will allow us to build out/test workflow in advance of your upcoming DAPR WF Component.

Please let me know if this makes sense or not.

Respectfully, John

PS: I can demo this for you, should you wish to see it working in action.

jkears commented 2 years ago

Hi @cgillum,

We completed the code-generated integration and your current WF API fits perfect within our framework.

We still can't test it as we still require the ability to run durabletask-sidecar (image) within it's own service that can be spun up within the Tye DAPR services.

If you could be so kind as to provide the source used to create the durabletask-sidecar image, so that we can complete integration testing that would be fantastic.

Also, how do we get the event data? The current API does not seemingly provide us access to data.

For example, we publish this way...

image

Where on line 53, we are publishing the command as the event by the same name, and where the event payload is the command, and thusly, how within the actual orchestration logic would I gain access to the command that was passed in?

image

I am assuming that there would be an override where you could specify any Event Payload type on the WaitForExternalEventAsync API such as

`WaitForExternalEventAsync<EventPayloadType?>("DoSomething2Command", TimeSpan...

jkears commented 2 years ago

Hi @cgillum

I apologize for this reminder to you, however would you please provide the source used to create the POC image so that we can complete testing workflow integrations using the existing POC approach.

Having the image source code would allow us to create an interim host service that we can run along side our existing domain services (via TYE).

This would allow us to test Durable Task-based workflows/SAGAs integration within our existing services, and thus be ready once the final DAPR Workflow component becomes available.

As we are not GO developers (C# only), we unfortunately cannot be of much assistance in the development of the new Workflow component, however we if we are able to integrate with the POC approach such that we build a bunch of workflows/SAGAs, from which we could provide you with additional thoughts, ideas and feedback for consideration within the final GO version of the DAPR Workflow component.

Please let me know if you would our time to test your preview versions of the GO Workflow component (as it becomes available).

Please let me know if you will provide the source code for the POC image.

Thanks John

cgillum commented 2 years ago

So you're saying that Tye doesn't work with existing sidecar images, and that they need to be built as part of the Tye project? I haven't tried using Tye yet, but that seems counterintuitive to me. The prebuilt image is publicly available on Docker Hub with the name/tag cgillum/durabletask-sidecar:0.3.3-dapr.

I went ahead and sent you an invite to the private repo that contains the POC sidecar source code.

Also, how do we get the event data? The current API does not seemingly provide us access to data.

Ah, it looks like I didn't create an API for actually processing the event data from within the workflow. One hacky solution would be to just case the Task from WaitForExternalEvent into a Task<object>, but that's obviously pretty hacky. I can do an update that includes an overload for getting the data out.

UPDATE: I pushed a WaitForExternalEvent overload you can use for getting the data out: https://github.com/cgillum/durabletask-dapr/commit/99aa3dbd87c57e9939125eda352742083f3dd336.

jkears commented 2 years ago

Hi @cgillum ,

Thank you for all of your support!

I am able to create new orchestrations, invoke events (commands per orchestrations step) and terminate orchestrations when I run the POC sidecar project in VS (debug). I am not sure why that works, while the image approach did not, but either way I am super happy!

I would love to show (you via Zoom) how well this all works, plus discuss how we can become beta-testers of the DAPR Workflow component block.

Cheers John