API for custom apps that like to produce events (CloudEvents) in easy, secure and tracable way

[cardil]

Problem I feel like Knative Eventing doesn't quite address most common use case. That use case would be to write custom business app that like to produce events.

Let's imagine for example a use case for a banking company. We have a "MyBank" that would like to use Knative as primary deployment model. They would like to create a couple (hundred :-P) microservices, that will focus on their business cases.

Let's take 3 as an example: credits, accounts, cards - each of them will be responsible for managing respective area. Each o those services will be built as REST/GraphQL Knative Service, and deployed on K8s. This is awesome - Knative Serving will directly suited for that example.

Let's now imagine those 3 services would like to consume events. Awesome. Knative Eventing is for the task a good match. They just need to specify an endpoint that will receive Knative Eventing events.

What we provide if those services would like to publish events as well? Not much! We can tell only:

Feed your events to a broker

That's the problem. Broker is internal part of Eventing. Also, it's a cluster administrator specific resource. How developer that writes this custom app, will know where to send their events? Event if delivered by a configuration, still it's messy to ask our users to use it directly. By the way, currently, it's implementation is changing.

There is also another problem linked to using Broker directly. Visibility and security. We have no way of telling which service is sending what events, and if it even should. It's different for Knative event sources, as they are created by cluster administrator, or it's approval. Also, event sources defines event types, source. It's traceable. We can, and we do, graph them to visualize app's topology. None of that is possible, if we let users feed events to Broker directly.

Proposition

I propose to develop a simple API that developers can use to send custom events.

See refined proposition below: https://github.com/knative/eventing/issues/3186#issuecomment-632662449

To start discussion I propose creating a Tap type (mirroring Sink). It will define event type that can be sent, and their source. This type will also have a .Status.Key field, that contain generated (or provided by an administrator), key. This key should be used to sign events. Signing algorithm should be configurable. The .Status.GatewayAddress field will contain a URL that developer should use to send events, decoupling it from Broker, and it's implementation.

That will let us change Broker to accept only trusted events, and let up visualize app's topology displaying an event route.

Below are examples of pseudo code of such resources. Specifying those by cluster administrator will let accounts service send events of possible fraud detection and that funds are depleting on one of custom accounts. Of course, cluster administrator would need to forward Tap's status values to account as ConfigMap.

---
apiVersion: sources.knative.dev/v1alpha1
kind: Tap
metadata:
  name: accounts-possible-fraud-detected
  namespace: demo
spec:
  type: com.mybank.events.PossibleFraudDetected
  source: //mybank/accounts
  signingAlgorithm: xor
status:
  gatewayAddress: http://default-broker.demo.svc.cluster.local
  key: Yt31e4zw+3ymmzVrCcmjFIJludLvV00pI6bUP6lQ/g3noBIEFRTHEgKoA84Kc5Cv
---
apiVersion: sources.knative.dev/v1alpha1
kind: Tap
metadata:
  name: accounts-funds-depleted
  namespace: demo
spec:
  type: com.mybank.events.FundsDepleted
  source: //mybank/accounts
  signingAlgorithm: rsa
status:
  gatewayAddress: http://default-broker.demo.svc.cluster.local
  key: 0ozeHGTl9kes1RwC7+bgx8wktKv10YTGly+ViHZbEH3UTziJc1hTzxU+/4KIEYgJ

Persona: Event producer

Exit Criteria Have a simple, but secure API for custom apps that would like to publish events. Decouple users from using Broker directly, and let trace event propagation and secure event producers.

Time Estimate (optional): 15d

[jchesterpivotal]

As an immediate bit of bikeshedding, Tap may cause confusion with the Wire Tap pattern. Perhaps Faucet, if you're looking for a hydraulic theme?

[evankanderson]

Have you looked at the CloudEvents SDKs here? https://github.com/cloudevents/

For constructing the event payload and envelope via code, I'd suggest contributing to the CloudEvents libraries rather than duplicating them in Knative.

In terms of configuring the destination, have you looked at SinkBinding as a way to supply (a single) destination for events to the application? More documentation on SinkBinding as a mechanism for combining a Deployment + addressing information is here.

[cardil]

Have you looked at the CloudEvents SDKs here? https://github.com/cloudevents/

Of course. This assumes you would like to send custom CloudEvents. I should have stated that.

In terms of configuring the destination, have you looked at SinkBinding as a way to supply (a single) destination for events to the application? More documentation on SinkBinding as a mechanism for combining a Deployment + addressing information is here.

This is not about destination. This is about source.

[vaikas]

Could you clarify little more about this?

What we provide if those services would like to publish events as well? Not much! We can tell only:

Feed your events to a broker

That's the problem. Broker is internal part of Eventing

I don't understand what you mean by the internal part? https://github.com/knative/eventing/blob/master/docs/spec/broker.md

For the security there's this: https://github.com/knative/eventing/issues/2273

For visibility, there was talk and should be an issue (that I can't find right now) about being able to control what events are allowed to be sent to Broker. I know @csantanapr was involved in this discussion for example, do you recall the status of that?

[evankanderson]

Feed your events to a broker

That's the problem. Broker is internal part of Eventing. Also, it's a cluster administrator specific resource. How developer that writes this custom app, will know where to send their events? Event if delivered by a configuration, still it's messy to ask our users to use it directly. By the way, currently, it's implementation is changing.

Broker is intended to be part of the external surface area of eventing. The purpose of having "a bucket of events that you can filter by attribute" is to make it easy for event producers to send events -- they should go into the bucket. Most of the time, there should be just one bucket (per namespace), but there are good cases for allowing multiple buckets, so that's why you can create multiple Brokers (one of the first examples was PII and non-PII Brokers).

Unfortunately, it's clear that we haven't done a good job documenting this, because it confused you. That's definitely on us. It's also taking longer than we'd have liked to build the infrastructure -- we started talking about the idea in Dec 2018 after Kubecon Seattle (in a room with about 30 of us), but we didn't get the first implementation out until mid 2019, and that implementation wasn't very efficient (the goal was to test out the API interface, and replace the implementation if it worked). Fast-forward to today, and the multi-tenant Broker is (I hope) close to meeting our initial goals in Dec 2018.

For the record, we talked about 3-4 scenarios in that 2018 meeting:

1. I just want to publish events. You need some way to know where to send them (SinkBinding), plus the Broker to perform routing.

2. I just want to consume events like X, I don't care how they are published. This is the use case for Trigger, which it sounds like you were able to find fairly clearly.

3. I want to transform events through a series of steps. This is the use-case for Channels / Subscriptions and Sequence.

4. I want to perform complex orchestration of events. This is something that systems like OpenWhisk Composer and AWS Step Functions perform. We don't have a built-in system for this today, though you can build one using the primitives like Channel and the Parallel / Sequence flows.

There is also another problem linked to using Broker directly. Visibility and security. We have no way of telling which service is sending what events, and if it even should. It's different for Knative event sources, as they are created by cluster administrator, or it's approval. Also, event sources defines event types, source. It's traceable. We can, and we do, graph them to visualize app's topology. None of that is possible, if we let users feed events to Broker directly.

Agreed that security is an issue that Knative hasn't addressed yet. There was a somewhat-promising proposal earlier this year that ended up dead-ending in Istio-specific implementation, but #2272, #2273 and #705 have a high-level sketch of the approaches we've been considering. In particular, if you have specific security requirements, adding them to #2272 would be extremely helpful.

[evankanderson]

If the details from https://github.com/knative/eventing/issues/3186#issuecomment-632250524 were helpful, would it be useful to add them to the Knative Docs eventing landing page? I wouldn't expect eventing consumers to have paid attention or found any of the notes from the December meeting two years ago.

[cardil]

Reply to @vaikas

I don't understand what you mean by the internal part? https://github.com/knative/eventing/blob/master/docs/spec/broker.md

I treated Broker as internal part. Mostly due to fact it accepts all events without authentication. Also, the name suggest that, as Brokers are the people that operate stocks. You can send them request, but often it's fronted with some other infrastructure like IT system, secretary, etc. Just to keep their minds focus on their main job - stocks, not outside human interaction. @evankanderson comment above explained that to me. I think such paragraph should be in docs!

For the security there's this:

2273

If so, the broker being intended to be public API, having no authentication and traceability is a Huge security hole, and architectural mishap. Serving don't pose a threat of security for custom apps as it's transparent for client data. Eventing, with this open up broker, forces business users to let go of security and accountability for their events! Keep in mind that if user sends events to Kafka, Service Bus, or JMS, with respective native client library, they surly need to authenticate, before doing so.

That's why I posted this issue to prepare an easy, secure, traceable, frontend for Broker. At the same time letting us secure internal communication.

For visibility, there was talk and should be an issue [..]

Visibility, accountability, traceability, whatever we call it, is as important as security for most regulated industries like financial, medical etc. Again, open up Broker, as a design is a huge omission.

We should be in position to boldly respond to those most regulated industries:

Yes. Knative Eventing is secure and traceable. You could rely on it to transfer your most valueable assets via it!

[cardil]

Reply to @evankanderson

Broker is intended to be part of the external surface area of eventing. The purpose of having "a bucket of events that you can filter by attribute" is to make it easy for event producers to send events -- they should go into the bucket. Most of the time, there should be just one bucket (per namespace), but there are good cases for allowing multiple buckets, so that's why you can create multiple Brokers (one of the first examples was PII and non-PII Brokers).

Thanks for explanation. This paragraph surly be included in Broker docs.

Agreed that security is an issue that Knative hasn't addressed yet. There was a somewhat-promising proposal earlier this year that ended up dead-ending in Istio-specific implementation, but #2272, #2273 and #705 have a high-level sketch of the approaches we've been considering. In particular, if you have specific security requirements, adding them to #2272 would be extremely helpful.

Thanks for referencing those.

[cardil]

Rethinking my proposition after comments I stand down with it as currently Broker is wide open. But, I would like to change my proposition a little.

I propose creating a FaucetBinding type (mirroring SinkBinding). It will define event type that can be sent, and their source. We can also specify signing algorithm. User must sign events while sending them to prove oneself indisputably. After creating this type, eventing system will reconcile it and creating a ConfigMap containing URL to send events, and Secret with keys to sign events with. Cluster Administrator can then mount those ConfigMap and Secret to some pods, enabling developers to use those values at runtime. That automatic creation of those resources is to ease of use for Cluster Admin and Developers.

Broker will be available only via TLS http, making sure that communication can't be intercepted. Broker will also accept only trusted events, by validating signature. Also, FaucetBinding information let us visualize apps topology displaying an event route.

---
apiVersion: sources.knative.dev/v1alpha1
kind: FaucetBinding
metadata:
  name: accounts-possible-fraud-detected
  namespace: demo
spec:
  type: com.mybank.events.PossibleFraudDetected
  source: //mybank/accounts
  signingAlgorithm: xor
  broker: default
---
apiVersion: sources.knative.dev/v1alpha1
kind: FaucetBinding
metadata:
  name: accounts-funds-depleted
  namespace: demo
spec:
  type: com.mybank.events.FundsDepleted
  source: //mybank/accounts
  signingAlgorithm: rsa
  broker: other

Reconciled resources are:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: accounts-possible-fraud-detected-event-gateway
  namespace: demo
data:
  gatewayAddress: https://default-broker.demo.svc.cluster.local
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: accounts-funds-depleted-event-gateway
  namespace: demo
data:
  gatewayAddress: https://other-broker.demo.svc.cluster.local
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: accounts-possible-fraud-detected-event-signing
  namespace: demo
stringData:
  key: 0ozeHGTl9kes1RwC7+bgx8wktKv10YTGly+ViHZbEH3UTziJc1hTzxU+/4KIEYgJ
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: accounts-funds-depleted-event-signing
  namespace: demo
stringData:
  cert: |
  -----BEGIN CERTIFICATE-----
  MIIEMzCCAxugAwIBAgIQU9ETHSLLW3VYHI1HZyrJyDANBgkqhkiG9w0BAQsFADCB
  myY52J8kYbHlP/Mx5PuH/eun1jtY01I=
  -----END CERTIFICATE-----
  key: |
  -----BEGIN RSA PRIVATE KEY-----
  MIIEpAIBAAKCAQEAyctl6QN5vvkBrMMJz4J9BnLCrNArgxpXJWqVo0sawSZhS/TA
  NFxE4XaDx8sX9/ZIJHi0Xm+cVmxBZSVi00E+l0MERM36Z4uze1oMg==
  -----END RSA PRIVATE KEY-----

[evankanderson]

For the security there's this:

2273

If so, the broker being intended to be public API, having no authentication and traceability is a Huge security hole, and architectural mishap. Serving don't pose a threat of security for custom apps as it's transparent for client data. Eventing, with this open up broker, forces business users to let go of security and accountability for their events! Keep in mind that if user sends events to Kafka, Service Bus, or JMS, with respective native client library, they surly need to authenticate, before doing so.

We're aware that lacking authentication and authorization primitives is big gap. There are some environments where this can be mitigated in other ways -- network security policies, service mesh functionality, or "not that important" data are some of the cases we've seen.

There's definitely other workloads where this is a critical feature gap, and I think it's well within the current architecture to add policy enforcement at critical places in the infrastructure. Closing this feature hole has been prioritized behind other efforts historically for three reasons:

1.  Defining the user-facing API for receiving events has been the top priority. Once the API (both K8s objects and data path interfaces) have been defined, it becomes possible to parallelize other work around tooling, abstractions, performance, etc. The recent v1beta work suggests this work is starting to come to fruition.

2. Getting a working data plane for customers is also critical. This has meant working with users' desired storage systems (see the multiple Channel abstractions), which took about a year to stabilize, and further work by the event delivery WG to provide reliable delivery and failure handling.

3. Uncertain requirements. Now that the above two priorities have reached the point that Eventing can be useful and recommended for some purposes, the next step is to highlight use cases where there are missing features, and figure out the best way to implement them. #2272 is where we're attempting to collect those cases -- as a simple test, would the ability to set ACLs based on Service Account identity (OIDC JWT) and Cloud Events message type be sufficient for your needs, or do you need to be able to apply finer-grained logic (for example, only accept certain subjects from certain service accounts)?

Rethinking my proposition after comments I stand down with it as currently Broker is wide open. But, I would like to change my proposition a little.

I agree that this is a gap, but I think this is a feature gap which can be addressed in an additive way, rather than needing to rebuild the Broker.

I propose creating a FaucetBinding type (mirroring SinkBinding). It will define event type that can be sent, and their source. We can also specify signing algorithm. User must sign events while sending them to prove oneself indisputably. After creating this type, eventing system will reconcile it and creating a ConfigMap containing URL to send events, and Secret with keys to sign events with. Cluster Administrator can then mount those ConfigMap and Secret to some pods, enabling developers to use those values at runtime. That automatic creation of those resources is to ease of use for Cluster Admin and Developers.

You may want to engage with the CloudEvents CNCF working group on event signing, if that's a requirement or of interest to you. There are some interesting challenges in event signing, both around key distribution and management, and on compatibility between signatures and middleware functionality like updating trace headers.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[n3wscott]

/remove-lifecycle stale

This might be a better project to explore in knative-sandbox vs core eventing.

[lberk]

@cardil given the movement/features added in knative since may, specifically tracing, has anything changed to the scope of this issue/request?

[grantr]

@cardil, I'm closing this so you can open a new issue with the remaining work after the discussion. Thanks for bringing this up!