conduktor-console container does not automatically configure Kafka Connector when KAFKACONNECTS environment variables are provided

[dylanjfeehan]

I'm providing:

CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: "zorp-zorp" CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: "My Local Connector" CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: "http://connect:8081"

to my conduktor-console image.

Where http://connect:8081 is the location of my kafka connector, which is running as a docker compose service named connect. I know that this endpoint is healthy because when i spin up conduktor-console, I can add a connector using this URL.

When I provide CDK_CLUSTERS_0_SCHEMAREGISTRY_URL, and i spin up the conduktor-console container, the schema registry is automatically connected. I expect the same behavior out of the KAFKACONNECTS property. Otherwise, what's the point in including these environment values?

https://github.com/conduktor/kafka-stack-docker-compose/blob/master/conduktor.yml URL and NAME are included here

https://docs.conduktor.io/platform/get-started/configuration/env-variables/#kafka-connect-properties ID and URL are listed here as properties

[Navirash]

Hello, I have also the same problem. Do you find a solution to fix ?

[AurelieMarcuzzo]

Hi,

Thanks for having created this issue.

Could you please tell me what is the exact behaviour on your side? I'm searching for logs saying that something is missing in the configuration, or a connection issue.

On my side, when I deploy Console 1.21.1 using these environment variables, everything works as expected:

CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: "http://schemaregistry:8085"
CDK_CLUSTERS_0_SCHEMAREGISTRY_PROPERTIES: "basic.auth.credentials.source=USER_INFO\nbasic.auth.user.info=admin-sr:secret"
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: "My Local Connect"
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: "http://connect:8083"
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_USERNAME: "admin-connect"
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_PASSWORD: "secret"

[Navirash]

Hi @AurelieMarcuzzo

Thank you for your answer. I also tried with version 1.21.1. I still don't see the Kafka Connects on the console. When I check the logs, I get a warning: WARN i.c.a.s.OnPremInitializationService - Cannot sync Kafka connect clusters because at least one of them is missing an ID

Here is my configuration :

extraEnvVars:
    - name: CDK_CLUSTERS_0_ID
      value: kafka
    - name: CDK_CLUSTERS_0_NAME
      value: kafka
    - name: CDK_CLUSTERS_0_BOOTSTRAPSERVERS
      value: bootstrap.kafka.svc:9092
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_ID
      value: "mq-connect"
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME
      value: "mq-connect"
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_URL
      value: "http://connect-mq-connect.svc.cluster.local:8083"

To deploy the Conduktor console, I use Helm

[AurelieMarcuzzo]

Hi @Navirash,

Thanks for the logs line, I think your extraEnvVars isn't in the right location in your values. Can you confirm it looks like:

config:
  organization:
    name: "Test"

  admin:
    email: "admin@conduktor.io"
    password: "admin"

  database:
    host: postgresql.conduktor
    port: 5432
    username: postgres
    password: conduktor
    name: conduktor

platform:
  extraEnvVars:
    - name: CDK_CLUSTERS_0_ID
      value: kafka
    - name: CDK_CLUSTERS_0_NAME
      value: kafka
    - name: CDK_CLUSTERS_0_BOOTSTRAPSERVERS
      value: bootstrap.kafka.svc:9092
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_ID
      value: "mq-connect"
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME
      value: "mq-connect"
    - name: CDK_CLUSTERS_0_KAFKACONNECTS_0_URL
      value: "http://connect-mq-connect.svc.cluster.local:8083"

You can find the exact paths detailed here: https://github.com/conduktor/conduktor-public-charts/blob/main/charts/console/README.md

[Navirash]

Hi @AurelieMarcuzzo I confirm to you that I have the same configuration. (values.yaml)

I also check the config.yml inside the container :

organization:
  name: test
clusters:
- id: kafka
  name: kafka
  bootstrapServers:bootstrap.kafka.svc:9092
  labels: {}
  ignoreUntrustedCertificate: false
  kafkaConnects:
  - url: http://connect-mq-connect.svc.cluster.local:8083
    name: mq-connect
    ignoreUntrustedCertificate: false

Navirash

[AurelieMarcuzzo]

Hi @dylanjfeehan,

We troubleshooted this with @Navirash, and it looks like a DB cleanup of the clusters solved the issue. I think that there are some residues from previous deployments, and that the Kafka Connect isn't updated when you upgrade the deployment. My recommendation would be either to purge the cluster DB and re-deploy Console, or to use the UI or the API directly.

Thank you very much.