Publish/Subscribe Dataplane based on Kafka

[lgblaumeiser]

Overview

Explain the topic in 2 sentences

Add a new dataplane exchange mechanism that is based on Kafka and provides a publish subscribe continuous data push from the provider to the consumer.

What's the benefit?

Data exchange which allows to continuously provide event based data. This enables a complete new world of use cases that are currently only supported with weird workarounds.

What are the Risks/Dependencies ?

None, as it is a feature addition

Detailed explanation

Current implementation

None

Proposed improvements

Based on a provider side Kafka installation, available topics are provided as assets in an EDC. After getting a contract, the Transfer Process metaphor is used to enable and disable access to the topic. The access is provided by a token that the consumer can use to directly enable the access to the topic in the available Kafka service.

Feature Team

Contributor

• Cofinity-X

Committer

• Andrea Bertagnolli

User Stories

• tbd

Acceptance Criteria

• [ ] A dataplane exists that provides a Kafka based publish subscribe data exchange
• [ ] The dataplane is quality assured with appropriate tests
• [ ] The relevance for a standard change is evaluated and a potential change is triggered

Test Cases

Test Case 1

Do a Kafka data transfer

Steps

1. An asset providing a Kafka topic as offering is created
2. A consumer negotiates for the asset and agrees to a contract
3. With the provided access data, the consumer access the topic on the provided Kafka instance

Expected Result

1. The consumer can access events on the Kafka topic
2. Based on the transfer process state, the access is provided or blocked. The consumer can reestablish a data transfer by changing the state of the transfer process (suspend/resume, reinitiate the transfer)

Architectural Relevance

• [X] This feature aligns with our current architectural guidelines
  • Data Sovereignty: All data sharing activities across company boundaries follow the Catena-X Regulatory Framework, in particular the Data Exchange Governance, and the Dataspace Protocol via a compliant Connector (like the tractusx-edc or similar, see Connector KIT)
  • [X] CX-0010 Business Partner Number
  • [X] CX-0013 Identity of Member Companies
  • [X] CX-0018 Data Space Connectivity (EDC)
  • [X] CX-0049 DID Document Schema
  • [X] CX-0050 Framework Agreement Credential
  • [X] CX-0149 Verified Company Identity
  • Interoperability: Digital Twins are used (compliant to the Digital Twin KIT and the Industry Core KIT)
  • [X] CX-0001 EDC Discovery API
  • [X] CX-0002 Digital Twins in Catena-X
  • [X] CX-0018 Data Space Connectivity (EDC)
  • Data Format:
  • [X] The data model is based on a published Semantic Model
• [ ] The impact on the overall system architecture has been assessed. The Feature does not require changes to the architecture or any existing standard? Please have a look here on the overarching architecture
• [X] Potential risks or conflicts with existing architecture has been assessed

Justification: (Fill this out, if at least one of the checkboxes above cannot be ticked. Contact the Architecture Management Committee to get an approval for the justification)

The impact is, that there is a new mechanism for data exchange. The impact on CX-0018 has to be evaluated. It has no real impact on the architecture except for an additional feature.

Additional information

• [X] I am aware that my request may not be developed if no developer can be found for it. I'll try to contribute a developer (bring your own developer)