Deployments do not pick up the configuration of Stream objects

Deployments do not pick up the configuration of Streams but use the default configuration.

You can reproduce this as follows (assuming that your DataCater installation is available under http://localhost:8080):

Create two streams for a remote Kafka cluster, e.g., on Confluent Cloud:

curl http://localhost:8080/api/alpha/streams \
-XPOST \
-H'Authorization:Bearer YOUR_TOKEN' \
-H'Content-Type:application/json' \
-d'{"name":"test","spec":{"kafka":{"topic":{"config":{}},"bootstrap.servers":"$HOST_OF_eREMOTE_KAFKA_BROKER"},"kind":"KAFKA"}}'

Create a pipeline that uses these streams as stream-in and stream-out:

curl http://localhost:8080/api/alpha/pipelines/ \
-XPOST \
-H'Authorization:Bearer YOUR_TOKEN' \
-H'Content-Type:application/json' \
-d'{"metadata":{"stream-in":"$UUID_OF_FIRST_STREAM","stream-out":"$UUID_OF_SECOND_STREAM"},"spec":{},"name":"test"}'

Create a deployment:

curl http://localhost:8080/api/alpha/deployments/ \
-XPOST \
-H'Authorization:Bearer YOUR_TOKEN' \
-H'Content-Type:application/json' \
-d'{"spec":{"pipeline":"$UUID_OF_PIPELINE"}}'

Investigate the deployment and note both DATACATER_STREAM-IN_CONFIG and DATACATER_STREAM-OUT_CONFIG:

kubectl describe pod datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf-bgskd                                                                    14:41:18
Name:             datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf-bgskd
Namespace:        default
Priority:         0
Service Account:  default
Node:             minikube/192.168.58.2
Start Time:       Wed, 16 Nov 2022 14:27:09 +0100
Labels:           datacater.io/app=datacater-pipeline
              datacater.io/pipeline=1
              datacater.io/revision=1
              datacater.io/uuid=0a37998b-9f49-4bc1-9c4c-72d23b03aabf
              pod-template-hash=86554c87f
Annotations:      <none>
Status:           Running
IP:               172.17.0.5
IPs:
IP:           172.17.0.5
Controlled By:  ReplicaSet/datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf-86554c87f
Containers:
datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf:
Container ID:   docker://d6822e67b31b36d83098deac557220a579ad6f56a862d0d1446a87cd30b6a7cc
Image:          datacater/pipeline:alpha-20221108
Image ID:       docker-pullable://datacater/pipeline@sha256:92a2aa7c2247b1c21891bd3f054611299d0d4f7aa912eb89bc870727e2b9ecf0
Port:           <none>
Host Port:      <none>
State:          Running
  Started:      Wed, 16 Nov 2022 14:27:25 +0100
Ready:          True
Restart Count:  0
Limits:
  memory:  4Gi
Requests:
  cpu:     100m
  memory:  1536Mi
Environment:
  MP_MESSAGING_OUTGOING_STREAM-OUT_TOPIC:  test
  MP_MESSAGING_INCOMING_STREAM-IN_TOPIC:   foobar
  DATACATER_STREAM-IN_CONFIG:              {key.deserializer=io.datacater.core.serde.JsonDeserializer, value.deserializer=io.datacater.core.serde.JsonDeserializer, bootstrap.servers=localhost:9092}
  DATACATER_STREAM-OUT_CONFIG:             {value.serializer=io.datacater.core.serde.JsonSerializer, bootstrap.servers=localhost:9092, key.serializer=io.datacater.core.serde.JsonSerializer}
Mounts:
  /usr/app from datacater-volume-0a37998b-9f49-4bc1-9c4c-72d23b03aabf (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-27675 (ro)
python-runner:
Container ID:   docker://23795a972c9f46781176127171e1a99758458495a632a188f5e722e7efa37a66
Image:          datacater/python-runner:alpha-20221116
Image ID:       docker-pullable://datacater/python-runner@sha256:a346441219021abc2db2f96404d4812b6e37cdd105d68746b5719535e708e01f
Port:           50000/TCP
Host Port:      0/TCP
State:          Running
  Started:      Wed, 16 Nov 2022 14:27:25 +0100
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-27675 (ro)
Conditions:
Type              Status
Initialized       True
Ready             True
ContainersReady   True
PodScheduled      True
Volumes:
datacater-volume-0a37998b-9f49-4bc1-9c4c-72d23b03aabf:
Type:      ConfigMap (a volume populated by a ConfigMap)
Name:      datacater-configmap-0a37998b-9f49-4bc1-9c4c-72d23b03aabf
Optional:  false
kube-api-access-27675:
Type:                    Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds:  3607
ConfigMapName:           kube-root-ca.crt
ConfigMapOptional:       <nil>
DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                         node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type     Reason       Age   From               Message
----     ------       ----  ----               -------
Normal   Scheduled    14m   default-scheduler  Successfully assigned default/datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf-bgskd to minikube
Warning  FailedMount  14m   kubelet            MountVolume.SetUp failed for volume "datacater-volume-0a37998b-9f49-4bc1-9c4c-72d23b03aabf" : failed to sync configmap cache: timed out waiting for the condition
Normal   Pulling      14m   kubelet            Pulling image "datacater/pipeline:alpha-20221108"
Normal   Pulled       14m   kubelet            Successfully pulled image "datacater/pipeline:alpha-20221108" in 13.622604465s
Normal   Created      14m   kubelet            Created container datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf
Normal   Started      14m   kubelet            Started container datacater-deployment-0a37998b-9f49-4bc1-9c4c-72d23b03aabf
Normal   Pulled       14m   kubelet            Container image "datacater/python-runner:alpha-20221116" already present on machine
Normal   Created      14m   kubelet            Created container python-runner
Normal   Started      14m   kubelet            Started container python-runner

DataCater / datacater

Deployments do not pick up the configuration of Stream objects #44