Add examples of using ChannelConnection Publishing

[przybylski]

Channel Connection Publishing is used by the third party providers to emit events to Eventarc. While this is similar to Custom Events is't using a different messages and API endpoints. So there is a benefit providing those samples to potential third party providers.

I did not test all of those end to end, so addiitonal testing might be needed.

[meteatamel]

Hi @przybylski, it looks like these samples and the existing custom events samples are almost identical, except the request and the actual publish calls are slightly different (eg. In C#, PublishChannelConnectionEventsRequest is sent with PublishChannelConnectionEventsAsync here instead of PublishEventsRequest and PublishEventsAsync in custom events).

Since the two flavours are almost identical, I'd like us to find a way to show both flavours in a single sample (so we don't have to maintain almost identical code in 2 separate places). But before we consider that, can you clarify why one would use PublishEvents vs. PublishChannelConnectionEvents? You mentioned the latter is for 3rd party providers but not clear what's exactly the difference is here in terms of the actual API call.

And do we really want samples for 3rd party providers? Our samples are usually linked from docs, is there a third-party doc somewhere that would use these samples, if we were to add them?

[przybylski]

@meteatamel you are correct that custom and third party publish are nearly identical, yet they are using different messages PublishEventsRequest VS PublishChannelConnectionEventsRequest, and different endpoint PublishEvents vs PublishChannelConnectionEvents, they also relate to a different resource channel vs channelConnection. So while the code is nearly identical I believe it's beneficial to have those extra samples.

As for sharing samples from the documentation, there has been some internal discussion with docs team that we could build a publicly available doc, rather than sharing docs on one by one basis. I will fill you in on the details during the week.

[meteatamel]

Ok but why would someone use one API/endpoint/message vs. the other? It's not clear from the APIs. On first look, PublishChannelConnectionEvents API sounds like something you call to send a connection event, but it looks like it accepts regular events.

In any case, I'd like us to see if we can handle both events & connection-events samples in a single sample with a flag or something, rather than 2 sets of separate samples.

[przybylski]

The client library is autogenerated from the service definition, so the function naming is directly inherited from the definition. Unfortunately, at the moment changing the function name would be a breaking change (both externally and internally).

The reason is also that they refer to a different resource type, which refers to AIP-121, meaning different manipulation methods for different resources. Additionally using method overload has internal consequences which are not addressable currently.

[meteatamel]

@przybylski, thanks for the explanation but these are implementation details that a regular user does/should not care about to be honest.

I think we need a crisp answer to this question: If I'm an Eventarc user who wants to publish some events to a channel, when would I use PublishEvents and when would I use PublishChannelConnectionEvents? What's the difference between the two from a user perspective?

Let me take a stab in answering (and please correct me if I'm wrong). PublishEvents is for users who want to publish to a custom channel (a channel not tied to a provider), whereas PublishChannelConnectionEvents is for users who want to publish to a third-party channel (a channel tied to a third-party provider). Is my understanding correct? (If so, I think it's unfortunate that we have two different methods PublishEvents and PublishChannelConnectionEvents for basically the same thing targeting different types of channels. I wish we found a way to combine the two or at least make them something like PublishCustomEvents and PublishThirdPartyEvents to make it more clear)

In any case, I'm still of the opinion that, we should combine the two sets of samples into a single set of samples and have a third-party flag to determine which method to call with what request. WDYT?

[przybylski]

There might be some confusion, which I hope I will be able to dispel here.

Individual customers (those who are interested in custom events) should be using PublishEvents method, as this is the API for direct interaction with the custom channel. Regardless if the channel is custom or not. However, individual customers COULD interact with non custom channel, but those events will be rejected, as the event needs to come from the third party events provider.

Keep in mind that custom and non custom channels are the same resource, but with a different configuration.

The documentation explains the channel as following

A channel is a resource through which a third-party provider can interact with a subscriber's project. The subscriber controls the channel, and the channel must be successfully connected so that it can receive and route events from the event provider to the subscriber. For details, see Third-party events in Eventarc.

Subscriber means individual client here, but subscriber is more generic nomenclature, especially when dealing with multiple actors (provider, subscriber, consumer)

PublishChannelConnectionEvents is the publishing API for the Channel Connection resource and cannot be user directly by the individual clients. The Channel Connection resource belongs to the third party events provider, only third party providers can crate and publish to the channel connection resource. However, at the end of the day, the event does end up in the individual client channel, but the level of indirection in publishing is required for various reasons.

Because of the above I would prefer to keep those examples separated, the channel one should be used directly by individual customers (for custom events), while the the channel connection one should be provider for the third party events provider.

Hope this makes sense.

[meteatamel]

Ok. I can't say I follow/agree all the nuances and it'll definitely be confusing to explain to users when to use what but it is what it is.

If I want to test these samples, how would I do it? In the previous samples, I simply created a channel (without a provider) and used that in my testing. In this case, I assume I need to have a third-party provider and create a channel from that provider, is that correct? I don't think I can use a provider like Datadog. I created a dummy provider in my project long time ago, I suppose I can use that but what would be the process to set this up for a regular user/third-party entity? I supposed they need to follow the process here to create a third-party provider and then go from there, right? https://cloud.google.com/eventarc/docs/third-parties/offer-source

[meteatamel]

Ok, I merged the PR and will refactor/move it to align with the other sample. I'll update here once that's done.

[meteatamel]

Ok, I did some slight refactoring and added these samples here: https://github.com/GoogleCloudPlatform/eventarc-samples/tree/main/publish-events/channel-connection

Moved the other samples here: https://github.com/GoogleCloudPlatform/eventarc-samples/tree/main/publish-events/channel

I couldn't test channel-connection samples end to end, as I didn't have a ready to use provider and project. I also didn't add detailed instructions on how to setup everything but marked those as TODOs for some other time.

If you have any feedback, please let me know. Otherwise, we can keep them as is now and try to improve them later.

[przybylski]

@meteatamel Here are instructions how to do end to end testing with dummy provider (using gcloud). First let's define some constants used in the test. dummy-provider - id of the provider dummy-publisher - project id of events publisher dummy-client - project id of individual client that is consuming events

1. Client creates a channel gcloud eventarc channels create test-channel --project dummy-client --provider dummy-provider --format=json This command will create a client channel bound to dummy provider. As an output a JSON object will be returned, that object will contain activationToken field, lets assume that its value is A

2. Events provider creates a channel connection gcloud eventarc channel-connections create test-connection --project dummy-publisher --channel projects/dummy-client/locations/$LOCATION/channels/test-channel --activation-token A This will create the channel connection that will route provider events to the client channel, note that the full name of the channel is being used.

3. (Optional) Check that the channel is now active gcloud eventarc channels describe test-channel --project dummy-client

4. Publish events to the channel connection gcloud eventarc channel-connection publish test-connection --project dummy-publisher --event-id $(uuidgen) --event-source '//dummy-provider/source' --event-type 'dummy-provider.v1.event' --event-data '{"key": "value"}' This will publish an event to channel connection which will end up in the client channel.

Obviously the above can be done in code using the client libraries (Java, C#) and publishing client libraries