The SDK needs gRPC service operations as Dapr pub/sub event subscribers.

Describe the proposal

Hello. I am working my way through the Dapr .NET SDK, writing C# code in Visual Studio (including using its debugger to attach to various processes involved) to use the various SDK operations to learn and see if this SDK will work for my needs. I'm done with State Management, which was quite easy thanks to the clear code in the SDK and examples. And have moved on to Pub/Sub.

In Pub/Sub I have completed pub/sub from a gRPC service to a webAPI controller. This was quite easy, again thanks to the clear SDK code and examples.

Now I want my pub/sub event subscriber to be another gRPC service operation. On close inspection of the code I do not see an immediate way to achieve setting up a subscription in C# like as done with the webAPI controller. Nor do I understand how to identify which gRPC service operation in the gRPC service should receive the published event. I have looked at appcallback.proto and some of that seems applicable, yet I do not see a clear way of setting up subscriptions to a specific gRPC service operation.

Please provide a clean way to setup one or more gRPC service operations as subscribers to a pub/sub event and then to dispatch the received event to the designated service operations. Publishing an event via gRPC service ops is nice and clean. That needs to be the case for subscribing and receiving events as well, and without writing much code to get the job done.

gRPC will be popular! And the .NET SDK needs to both publish and subscribe in gRPC services in a clean, low code manner in order to be attractive to Dapr customers.

Thanks, George

gRPC event subscriber possible features:

Use attributes on the gRPC service ops that subscribe to Dapr pub/sub in the same style that controllers do for HTTP. This approach provides a common customer (dev) experience across gRPC and HTTP. However there may be some down sides.
In service oriented systems it is common that a given service may have several different endpoints, e.g. queued endpoint, rpc endpoint, etc. Thus using this model, within a gRCP service project implement an HTTP controller that listens to the Dapr pub/sub http -- call it the Dapr pubsub http listener. This utilizes the existing WebAPI code to receive publications, including the use of attributes on controller methods. Then when a publication is received by the Dapr pubsub http listener contrioller, the controller method will instantiate an instance of the gRPC service (called service activation) and the invoke the proper gRPC service operation (called operation invocation). Note that service activation and operation invocation could be "pushed down" into a utility method called something like LaunchgRPCService(some args) so that the controller only has to call this single method to complete the pub sequence. Again, there may be downsides to this approach. However it is probably the "simplest thing that could possibly work" given the existing code, plus it does reuse a lot of the existing code. Plus this approach probably does not require the implementation of assembly scanning by reflection to instantiate a service instance and then invoke it.

I hope the above ideas get people thinking of various alternatives and their plusses and minuses.

Just to make my desired long term end results clear: Stepping back a bit from the above feature proposals, I would like to see all the grubby details of Dapr event subscription and dispatching pushed down into the ASP.NET Core framework, rather than have to write a bunch of service code to deal with these necessities or have to hard code event subscriptions via attributes. While attributes do have a place and make code easy to understand I would hope they would be optional with another alternative of programmatically subscribing to events available as well. I consider ASP.NET core framework extension methods as a reasonable way of pushing the details down into the framework.

Further I would like to see the end result of gRPC Dapr event subscription also be compatible with future releases of Dapr which will someday (I understand) support the dynamic definition of events by services, plus the ability to subscribe to these dynamically created events.

That said, my first 2 comments may contain useful information and perhaps short term solutions.

P.S. FYI, eShopOnContainers\src\BuildingBlocks\Services\Basket\Basket.API is an example of an ASP.NET Core web service that contains both Controllers for HTTP and a gRPC BasketService. So they can play together in the same web service to provide 2 different kinds of endpoints. Although that particular example does not use the HTTP endpoints for pub/sub but rather plain old HTTP interactions with clients.

gRPC will be popular! And the .NET SDK needs to both publish and subscribe in gRPC services in a clean, low code manner in order to be attractive to Dapr customers.

I think this is the north star - it would be awesome to have parity with how Daprized services are written for HTTP.

The main thing blocking this is that the Dapr runtime doesn't do server reflection. We need the runtime to be able to understand what endpoints are available as part of the server. Ultimately changes would be needed in the gRPC layer to be more similar to what we do for HTTP, so it could dispatch pub/sub to an endpoint you specify instead of a single generic endpoint.

/cc @yaron2 in case I'm missing anything

Hi Ryan,

Thanks for the notification. Please let me know if you received this email since I've never responded like this before and worry it might not get through to you.

This afternoon I finished cleaning up my TestNLearn Pub/Sub project using the .NET SDK. I used both the standard WebAPI controller approach and also the gRPC service approach that uses the AppCallback base class. Here are some of my insights.

The long term goal is, as you commented, " We need the runtime to be able to understand what endpoints are available as part of the server. Ultimately changes would be needed in the gRPC layer to be more similar to what we do for HTTP, so it could dispatch pub/sub to an endpoint you specify instead of a single generic endpoint. "

However, that will take time. Maybe months or longer. So as an interim solution below is an approach that would to use the existing AppCallback base, plus keep their code well organized.

First, separate the concerns of the service (code that implements the behavior required by the business logic) and the plumbing (aka infrastructure or iFX). For example, in my code I have a class named DaprServicesListener that inherits from AppCallback and which has no service code (business logic) in it. It is pure plumbing code, with the single responsibility to interact with Dapr and hand off the running of business logic to service code in a separate class. The DaprServicesListener for PubSub immediately calls a PubSubDispatcher static class method Dispatch(some appropriate args like topic, request, etc) that does 1) deserialization (via a separate Deserialize function that uses generic types (T) for the return type and is reusable across DTOs and services) and 2) once the deserialized .net DTO is returned it then instantiates (or uses a factory to do this or DI) an instance of IMyService and invokes the required service operation with the DTO as an arg. Then its done.

The above Deserialize function is plumbing code that can use generics and go into an iFX folder containing a few projects, like one to support Services. Using generics makes this code resueable across many DTOs. A code example of the PubSubDispatcher a can be provided as well, although to make that resuable across services one has to jump through some hoops that may be best left for more advance developers. If you want the details just ask. This means the developer would extend the boiler plate code of the PubSubDispatcher as necessary for their service ops and their DTOs as args. This is the big picture of an interim solution.

Specifically such an interim solution would furnish a DaprEnabledGrpcService template for a project with the above code in it, namely a generic Deserializer and a boilerplate example of the PubSubDispatcher. Note that the template could also include similar help for Service Invocation (which I start exploring tomorrow), and Binding (which I'll likely get to early next week).

While I've never made a project template before, I'll bet it would not take very long since it is just a modification of the existing gRPCService template. A template for a DaprEnabledGrpcService template might just be a low cost way to make using gRPC Services with Dapr easier for developers!

Thanks, George

On Wed, Dec 9, 2020 at 4:43 PM Ryan Nowak notifications@github.com wrote:

gRPC will be popular! And the .NET SDK needs to both publish and subscribe in gRPC services in a clean, low code manner in order to be attractive to Dapr customers.

I think this is the north star - it would be awesome to have parity with how Daprized services are written for HTTP.

The main thing blocking this is that the Dapr runtime doesn't do server reflection https://github.com/grpc/grpc/blob/master/doc/server-reflection.md. We need the runtime to be able to understand what endpoints are available as part of the server. Ultimately changes would be needed in the gRPC layer to be more similar to what we do for HTTP, so it could dispatch pub/sub to an endpoint you specify instead of a single generic endpoint.

/cc @yaron2 https://github.com/yaron2 in case I'm missing anything

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dapr/dotnet-sdk/issues/493#issuecomment-742081369, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRXHHIOUIXEOV77HMA42ALST7VPRANCNFSM4UJZKL7Q .

Thanks @georgestevens99

By the way if you don't like the email interface, you can always just browse to this issue and use the web UI. I don't think I've ever replied to a github email notification via email 🤔

First, separate the concerns of the service (code that implements the behavior required by the business logic) and the plumbing (aka infrastructure or iFX).

This is effectively the job that frameworks do - you provide a mapping some kind of mapping to URLs or topics, and the framework calls you. One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework. It's a big win for users when they can use something existing and importantly existing knowledge.

If you want the details just ask. This means the developer would extend the boiler plate code of the PubSubDispatcher as necessary for their service ops and their DTOs as args. This is the big picture of an interim solution.

I think what would be most helpful would be a link to the code publicly available somewhere if you have one.

What kind of next steps do you want?

Hi Ryan,

Thanks for your reply. For me it is a bit easier to respond to an email than navigate to git. I did visit the issue on git a while ago just out of curiosity.

You said " One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework ". That is true, and it applies as well to a project template for a visual studio (and vs code) for a daperized gRPC service. The Dapr team would have to weigh the ease of use their users would gain, vs the cost of docs, teaching, and support. I am getting along without these things, just guided by the code samples, so maybe good samples are the way to go right now. But gRPC is quite new, and newbies always appreciate things that speed them on their way to adoption.

You said " I think what would be most helpful would be a link to the code publicly available somewhere if you have one. ". That will be coming in a few days or so since it is only on my dev system right now and kind of a mess, as often exploratory code is. I will let you know when I make it public in my own Git repo.

I did read the gRPC Server Reflection Protocol document and it looks as if the implementation of that would furnish good metadata to use in informing the initialization of pubsubs, etc. This would reduce the need to code some of the Topic and Binding initialization data I suspect. Is implementing Server Reflection Protocol that in the long range plan?

I have found the code samples that come with the Dapr .NET SDK most helpful. Do you plan to produce code samples of the use of bindings with gRPC services?

Today I finished my example of Service Invocation, that goes along with my completed Pub/Sub and State examples. These are quite straightforward to use once one understands how. Your gRPC service code works well and is an impressive software accomplishment.

I am starting up on Bindings right now, and plan to use Azure Service Bus in my example as my "long haul" messaging backbone in addition to using the default Redis pubsub binding for local pub subs. Is the use of multiple pubsub bindings supported?

You said "What kind of next steps do you want?" The only thing that is of immediate concern is Binding examples, including the config that Dapr needs in the components (is that it?) directory. Or maybe it does not need a binding config file when it has the AppCallback furnishing the service with the requested data. Does it need both? -- The binding config file in components directory of dapr plus furnishing the AppCallback binding info?

So I will give it a shot with what I have and resort to Dapr Discord for assistance if I get stuck for a long time. There are very few code examples around on the internet using the Dapr .NET SDK, and even fewer that use Azure Service Bus and or that use gRPC services that I know of. There are a couple that use HTTP controllers to do pubsub things, though.

Thanks again for your response, George

On Mon, Dec 14, 2020 at 8:38 PM Ryan Nowak notifications@github.com wrote:

Thanks @georgestevens99 https://github.com/georgestevens99

By the way if you don't like the email interface, you can always just browse to this issue and use the web UI. I don't think I've ever replied to a github email notification via email 🤔

First, separate the concerns of the service (code that implements the behavior required by the business logic) and the plumbing (aka infrastructure or iFX).

This is effectively the job that frameworks do - you provide a mapping some kind of mapping to URLs or topics, and the framework calls you. One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework. It's a big win for users when they can use something existing and importantly existing knowledge.

If you want the details just ask. This means the developer would extend the boiler plate code of the PubSubDispatcher as necessary for their service ops and their DTOs as args. This is the big picture of an interim solution.

I think what would be most helpful would be a link to the code publicly available somewhere if you have one.

What kind of next steps do you want?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dapr/dotnet-sdk/issues/493#issuecomment-744978099, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRXHHI6BPT5A4UKZ5MLDJ3SU24XXANCNFSM4UJZKL7Q .

Hello, A quick read of the Dapr binding docs provides answers to my 2 questions:

" Does it need both?" = Yes. The items in the Dapr components directory are about providing capabilities in Dapr, while the AppCallback things in the gRPC service code are about identifying endpoints and other service oriented metadata to Dapr to aid it in delivering its capabilities.

And also my other question," Is the use of multiple pubsub bindings supported? " = Yes. This is achieved by the "PubSub" name used in the Dapr component matching that name in the AppCallback code in the gRPC service code.

However a code sample of using bindings, whether pub/sub or storage or whatever, in a gRPC Service would be quite helpful to developers coming up to speed on Daprized gRPC ASP.NET Core services.

Thanks, George

On Mon, Dec 14, 2020 at 9:27 PM George Stevens georgiageorge99@gmail.com wrote:

Hi Ryan,

Thanks for your reply. For me it is a bit easier to respond to an email than navigate to git. I did visit the issue on git a while ago just out of curiosity.

You said " One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework ". That is true, and it applies as well to a project template for a visual studio (and vs code) for a daperized gRPC service. The Dapr team would have to weigh the ease of use their users would gain, vs the cost of docs, teaching, and support. I am getting along without these things, just guided by the code samples, so maybe good samples are the way to go right now. But gRPC is quite new, and newbies always appreciate things that speed them on their way to adoption.

You said " I think what would be most helpful would be a link to the code publicly available somewhere if you have one. ". That will be coming in a few days or so since it is only on my dev system right now and kind of a mess, as often exploratory code is. I will let you know when I make it public in my own Git repo.

I did read the gRPC Server Reflection Protocol document and it looks as if the implementation of that would furnish good metadata to use in informing the initialization of pubsubs, etc. This would reduce the need to code some of the Topic and Binding initialization data I suspect. Is implementing Server Reflection Protocol that in the long range plan?

I have found the code samples that come with the Dapr .NET SDK most helpful. Do you plan to produce code samples of the use of bindings with gRPC services?

Today I finished my example of Service Invocation, that goes along with my completed Pub/Sub and State examples. These are quite straightforward to use once one understands how. Your gRPC service code works well and is an impressive software accomplishment.

I am starting up on Bindings right now, and plan to use Azure Service Bus in my example as my "long haul" messaging backbone in addition to using the default Redis pubsub binding for local pub subs. Is the use of multiple pubsub bindings supported?

You said "What kind of next steps do you want?" The only thing that is of immediate concern is Binding examples, including the config that Dapr needs in the components (is that it?) directory. Or maybe it does not need a binding config file when it has the AppCallback furnishing the service with the requested data. Does it need both? -- The binding config file in components directory of dapr plus furnishing the AppCallback binding info?

So I will give it a shot with what I have and resort to Dapr Discord for assistance if I get stuck for a long time. There are very few code examples around on the internet using the Dapr .NET SDK, and even fewer that use Azure Service Bus and or that use gRPC services that I know of. There are a couple that use HTTP controllers to do pubsub things, though.

Thanks again for your response, George

On Mon, Dec 14, 2020 at 8:38 PM Ryan Nowak notifications@github.com wrote:

Thanks @georgestevens99 https://github.com/georgestevens99

By the way if you don't like the email interface, you can always just browse to this issue and use the web UI. I don't think I've ever replied to a github email notification via email 🤔

First, separate the concerns of the service (code that implements the behavior required by the business logic) and the plumbing (aka infrastructure or iFX).

This is effectively the job that frameworks do - you provide a mapping some kind of mapping to URLs or topics, and the framework calls you. One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework. It's a big win for users when they can use something existing and importantly existing knowledge.

If you want the details just ask. This means the developer would extend the boiler plate code of the PubSubDispatcher as necessary for their service ops and their DTOs as args. This is the big picture of an interim solution.

I think what would be most helpful would be a link to the code publicly available somewhere if you have one.

What kind of next steps do you want?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dapr/dotnet-sdk/issues/493#issuecomment-744978099, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRXHHI6BPT5A4UKZ5MLDJ3SU24XXANCNFSM4UJZKL7Q .

Hello Ryan,I have just posted the code sample I mentioned previously in a public repo on git at https://github.com/georgestevens99/DaprNETSdk4RyanN Since I am aiming at breadth in my Dapr explorations, rather than depth, my comments in the code show how a dispatcher, and/or some of its parts can be broken out and generalized. This syncs with GRPC Server Reflection, which can make much of the work “automatic”! I will remove the code in a couple weeks, or when you tell me you have downloaded a version. I would rather it not be in public in the hacked up state it is now in. Thanks,George Sent from Mail for Windows 10 From: Ryan NowakSent: Monday, December 14, 2020 8:38 PMTo: dapr/dotnet-sdkCc: georgestevens99; MentionSubject: Re: [dapr/dotnet-sdk] The SDK needs gRPC service operations as Dapr pub/sub event subscribers. (#493) Thanks @georgestevens99By the way if you don't like the email interface, you can always just browse to this issue and use the web UI. I don't think I've ever replied to a github email notification via email 🤔First, separate the concerns of the service (code that implements the behavior required by the business logic) and the plumbing (aka infrastructure or iFX).This is effectively the job that frameworks do - you provide a mapping some kind of mapping to URLs or topics, and the framework calls you. One of the hesitations I have about building a framework into the Dapr SDK is that we would take on the burden for documenting, teaching people, and supporting the framework. It's a big win for users when they can use something existing and importantly existing knowledge.If you want the details just ask. This means the developer would extend the boiler plate code of the PubSubDispatcher as necessary for their service ops and their DTOs as args. This is the big picture of an interim solution.I think what would be most helpful would be a link to the code publicly available somewhere if you have one.What kind of next steps do you want?—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

dapr / dotnet-sdk

The SDK needs gRPC service operations as Dapr pub/sub event subscribers. #493

Describe the proposal