Correlation Use Cases

[duglin]

On the 8/30 call the topic of correlating events came up. This issue is meant to gather people's requirements and use cases for correlating events. Please add items as comments.

Some aspects of this that were brought up on the call:

• when to put this data in a CloudEvents property vs in the payload (data attribute)
• who will use this data and for what purpose? Meaning, can it be used for other purposes too?

[dmigliori]

Attached is a proposal for "Ontology-Based" CloudEvents which defines the attributes of the CloudEvents header within a Message object, with the "data" attribute comprising a common 2D-array format for multiple ontology-based contextual events, with each row representing a state of a single attribute of a single object (instance) of an ontology class. These transported events can be persisted/accumulated in a historic event store that provides correlation among objects of multiple domain ontologies.

This ontology-coupled event format is a subset of a reference architecture for cross-industry semantic interoperability. I've been actively advocating this format to consortia working groups for other event format standards including GS1 EPCIS, OMG DDS, and OASIS XDI.

Hopeful that this comment leads to a further discussion within a CloudEvents call. Ontology-Based CloudEvents.pdf

[erikerikson]

We are particularly interested in use cases that support identifying causal relationships between events.

Consider a series of events resulting from the detection of an object passing an array of sensors. Those events could be explicitly noted causal precursors to the emission of an event declaring the speed at which the object was observed passing. This, in turn, could be the precursor to the production of an event indicating a speed constraint violation. Processing a speed constraint violation could trigger a workflow recognizing the unique identifier (e.g. license plate) associated with the object and subsequently the legally responsible party. A final event (to this subset of the total workflow) could send a notice of fine to the registered mailing address of that responsible party and have as causal precursor the speed constraint violation event and unique identifier recognition event.

The causal chain represented in the declared correlation between events facilitates answering various useful questions that boil down to:

How the heck did this happen?

Causality between events becomes particularly and differently salient at the point that a system wants to query another system about the eventual consequence of an event.

Consider a person who submits money to a betting house wagering on the outcome of a horse race. They are given a receipt representing the successful delivery of the funds (i.e. their bet) and wants to submit that receipt as part of a query for the balance of their account once the bet is resolved (i.e. following the race's completion). The conclusion of the race could produce an event that once verified would cause the emission of an event updating the user's account balance as a result of that outcome and the additional causal precursor of the bet.

A more OLTP oriented scenario might include a read after write circumstance such as asking for the contents of a shopping bag after submitting an event indicating a desire to add an item to that bag. If that event is recorded in a persistent log (e.g. a Kafka topic) and the processor of that log mutates database record(s) representing the shopping bag, the API from which the writer of the event subsequently reads the record(s) will want to respond to the caller only after processing the original event. This would consist of ensuring the identified event was processed (i.e. its offset in the topic) before responding. Note that in this circumstance only one primal event exists and no causal chain between events is present. If, instead, the processor of the original event produced a secondary event that resulted in the mutation of interest to the shopping bag, the causal relationship between those events is necessary to identify the satisfaction of the precondition for responding to the original shopping bag contents request.

[duglin]

I am not interested in anything that will lead to me getting more speeding tickets!

Sent from my iPad

On Sep 18, 2018, at 8:12 PM, Erik Erikson notifications@github.com wrote:

We are particularly interested in use cases that support identifying causal relationships between events.

Consider a series of events resulting from the detection of an object passing an array of sensors. Those events could be explicitly noted causal precursors to the emission of an event declaring the speed at which the object was observed passing. This, in turn, could be the precursor to the production of an event indicating a speed constraint violation. Processing a speed constraint violation could trigger a workflow recognizing the unique identifier (e.g. license plate) associated with the object and subsequently the legally responsible party. A final event (to this subset of the total workflow) could send a notice of fine to the registered mailing address of that responsible party and have as causal precursor the speed constraint violation event and unique identifier recognition event.

The causal chain represented in the declared correlation between events facilitates answering various useful questions that boil down to:

How the heck did this happen?

Causality between events becomes particularly and differently salient at the point that a system wants to query another system about the eventual consequence of an event.

Consider a person who submits money to a betting house wagering on the outcome of a horse race. They are given a receipt representing the successful delivery of the funds (i.e. their bet) and wants to submit that receipt as part of a query for the balance of their account once the bet is resolved (i.e. following the race's completion). The conclusion of the race could produce an event that once verified would cause the emission of an event updating the user's account balance as a result of that outcome and the additional causal precursor of the bet.

A more OLTP oriented scenario might include a read after write circumstance such as asking for the contents of a shopping bag after submitting an event indicating a desire to add an item to that bag. If that event is recorded in a persistent log (e.g. a Kafka topic) and the processor of that log mutates database record(s) representing the shopping bag, the API from which the writer of the event subsequently reads the record(s) will want to respond to the caller only after processing the original event. This would consist of ensuring the identified event was processed (i.e. its offset in the topic) before responding. Note that in this circumstance only one primal event exists and no causal chain between events is present. If, instead, the processor of the original event produced a secondary event that resulted in the mutation of interest to the shopping bag, the causal relationship between those events is necessary to identify the satisfaction of the precondition for responding to the original shopping bag contents request.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[duglin]

What do people want to do with this issue? Based on our previous conversations we resolved to do correlation via extensions so I'm inclined to think we can close this one. I do however think that it might be useful to add some language into the primer (perhaps via examples) to help explain/elaborate on how we see people supporting those usecases. @cathyhongzhang I think you were thinking about doing something like this, no?

[cathyhongzhang]

OK, I will think about how to elaborate this.

[duglin]

@cathyhongzhang any more thoughts on this one? I'm going to suggest that we close this and we can reopen it later if needed. WDYT?

[cathyhongzhang]

That is fine with me. Actually people already added some workflow use cases which show the need for correlating one event to another event in the same serverless application workflow. Please refer to section "Use Case Examples" of https://github.com/cncf/wg-serverless/blob/master/workflow/spec/spec.md. Eric brought up some interesting use cases too. It will be great if Eric can add them to the workflow spec.

[duglin]

on the call today we agreed to close this issue - we can reopen if new info comes up.