search

bakdata / kpops

Deploy Kafka pipelines to Kubernetes
https://bakdata.github.io/kpops
MIT License
12 stars 1 forks source link

Reference output topics of components from other pipelines #513

Open philipp94831 opened 1 month ago

philipp94831 commented 1 month ago

Currently it is only possible to reference output topics of components within the same pipeline. It would be great to have a mechanism to also reference the output topic of anothers piplines components. A syntax I could imagine would be

- type: streams-app
  from:
    components:
      my-component:
        type: input
        pipeline: my/other/pipeline # path to pipeline from pipelines base dir

If pipeline is omited, the behavior does not change, i.e., a component within the same pipeline is referenced.