johnandersen777
commented
1 year ago We've started a discussion over here around announcements of federated CD events leveraging ActivityPub/Streams/Linked Data Notifications: https://codeberg.org/forgejo-contrib/discussions/issues/12
- Related
- https://github.com/cloudevents/spec/issues/830#issuecomment-923021758
- https://github.com/ietf-scitt/use-cases/issues/14
- RFCv3: IETF SCITT: Use Case: OpenSSF Metrics: activitypub extensions for security.txt
- https://github.com/cloudevents/spec/issues/1162#issuecomment-1447158952
- https://github.com/intel/dffml/issues/1315
- https://github.com/cdevents/spec/blob/main/spec.md#cdevents-custom-data
- https://github.com/opencontainers/distribution-spec/issues/388
- https://github.com/openvex/spec/issues/9
- https://github.com/guacsec/guac/issues/460
duglin
deissnerk
github-actions[bot]
@deissnerk and myself (and several of our respective coworkers) have been working through a number of scenarios in the last few months. There will be some PRs based on the outcome of this work, but I'd like to capture the resulting requirements for the group before we come around with solutions for them.
In Discovery, we have so far only really considered the side of the discovering party and how we can tell a "discoverer" what services (subscription managers) are available and what events can be subscribed from them.
As we as trying to enable event flow between two rather large platform ecosystems, we find that we als need to consider the side of the publishers and how they can "announce" to a middleware infrastructure that they have services and can raise events.
An exemplary use case is that there is an event publisher somewhere in a solution inside the SAP platform and there is a prospective event subscriber like an Azure Function inside an Azure platform subscription. We need to make it possible for that subscriber to make it known to the SAP event bus that it has a certain set of evets that it can raise and the SAP event bus must then be able to make it known to Azure Event Grid that such events may be sent.
Why do we need that information? At a minimum because there ought to be a drop-down box in the Azure portal where the developer can select that source and the kinds of events they want to subscribe to.
The present pull-style discovery model is insufficient for this. We need to be able to push the required discovery metadata along the future event flow path and get it near the prospective subscriber.
The fundamental assumption we are making here is that we establish trust at the platform level (that is a separate discussion that I will also file an issue for) and are flowing consolidated event streams between SAP and Azure in either direction. The (de-)multiplexing model we have allows for those event streams to be spliced up into per-tenant streams. That creates a flow path from a tenant scope inside of the SAP platform to a tenant scope inside the Azure platform.
It should then be possible for a custom application inside that tenant scope of the SAP platform to announce that it makes a certain set of events available for consumption. It basically announces the "service object" that we already have in the discovery API. That announcement should then be able to follow the established flow path and become available in the discovery API endpoint for the respective Azure tenant scope.
The announcement is an event. The "target" of such announcements is new endpoint on the discovery service that accepts such events and then decides whether to make them available to its callers. The relationship between the publisher and the discovery service may be direct, meaning that there is no subscription setup required to send the event. An implementation may also choose to subscribe for announcements at the publisher or the publisher's middleware. An announcement may end up being broadcast to several discovery services.
An announcement can take two forms: The publisher can announce on such a consolidated event path that there will be events of that sort coming and may then start sending them right away without requiring a subscription. For dispatching events out of that consolidated stream into particular endpoints such as a serverless function, there would still be a subscription manager that manages access to the consolidated stream. That approach has the downside of sending events through that consolidated channel that have no active subscribers, but the upside of avoiding the complex bookkeeping of the second approach:
The publisher may announce and the announcement will, as in the current service object, declare a subscription manager endpoint. When that announcement is then accepted by a discovery service, it might quite well be that while the discovery service can communicate with the publisher's subscription manager endpoint, prospective subscribers might not because of access permissions or network segmentation. In that case, the discovery service might host a subscription manager proxy. The subscription manager proxy is reachable by prospective subscribers "on the inside". The service objects that are returned by the discovery service then have an overridden subscription manager endpoint that points to the proxy. Each (un-)subscription made on the proxy then translates to the same operation on the upstream publisher. That model can then be chained. What makes that approach complex and potentially brittle is that intermediaries can't trust their subscribers to clean up subscriptions properly, which means that they have to do their own bookkeeping.