bakdata / kpops

Deploy Kafka pipelines to Kubernetes
https://bakdata.github.io/kpops
MIT License
12 stars 1 forks source link

Support referencing additional output topics from components #484

Open torbsto opened 2 months ago

torbsto commented 2 months ago

Let's say we have the following pipeline:

- type: my-producer
  to:
    topics:
      topic-with-role-1:
        role: role1
      topic-with-role-2:
        role: role2
- type: my-streams
  from:
    components:
      my-producer:
        type: input

This doesn't work because from components always uses the output topic and doesn't support output topics with roles. This could be for example changed by adding a new key called fromRole (role already exists if you want to use specify an additional input topic)

- type: my-producer
  to:
    topics:
      topic-with-role-1:
        role: role1
      topic-with-role-2:
        role: role2
- type: my-streams
  from:
    components:
      my-producer:
        type: input
        fromRole: role1