console (subchart): incorrectly sets zero valued connect configs

[arthurfedorov]

Hi everyone, first of all, i just want to say thanks for the cool service.

Deployed by helm chart Redpanda appVersion: v24.2.2 Redpanda console appVersion: v2.4.6

I have almost default values on it, except ingress values for console, and custom docker image for connectors. Dockerfile:

FROM redpandadata/connectors:v1.0.26

LABEL version=mongodb-kafka-connect:v1.0.5

COPY mongo-kafka-connect-1.13.0-all.jar /opt/kafka/redpanda-plugins/mongodb-kafka-connect/mongo-kafka-connect-1.13.0-all.jar

After helm install redpanda -n redpanda, everything is up

k get pods -n redpanda
NAME                                   READY   STATUS    RESTARTS       AGE
redpanda-0                             2/2     Running   0              118m
redpanda-1                             2/2     Running   0              118m
redpanda-2                             2/2     Running   0              118m
redpanda-connectors-65b5595784-wtqkt   1/1     Running   1 (118m ago)   118m
redpanda-console-8598665998-x9zww      1/1     Running   0              115m

When i open Connectors tab in Redpanda console, i see available connectors.

After I create any of the Mongodb connectors, it is created, but it is not in the interface. The logs in pod show the following error

│ {"level":"warn","ts":"2024-10-14T17:05:19.753Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}              │
│ {"level":"warn","ts":"2024-10-14T17:05:19.917Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}              │
│ {"level":"warn","ts":"2024-10-14T17:06:14.084Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}              │
│ {"level":"warn","ts":"2024-10-14T17:13:17.970Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}              │
│ {"level":"warn","ts":"2024-10-14T17:13:18.138Z","msg":"failed to list connectors from Kafka connect cluster","cluster_name":"connectors","cluster_address":"http://redpanda-connectors.re │
│ dpanda.svc.cluster.local:8083","error":"Get \"http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status\": context deadline exceeded"}   

But when I go into Redpanda console pod and do curl http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status, the list of connectors is visible.

/app $ curl http://redpanda-connectors.redpanda.svc.cluster.local:8083/connectors?expand=info&expand=status
/app $ {"mongodb-source-connector-xz6j":{"info":{"name":"mongodb-source-connector-xz6j","config":{"connector.class":"com.mongodb.kafka.connect.MongoSourceConnector","publish.full.document.only":"false","output.schema.infer.value":"false","startup.mode":"latest","topic.creation.default.partitions":"1","connection.password":"my_password","output.format.value":"json","publish.full.document.only.tombstone.on.delete":"false","connection.username":"my_user","output.format.key":"json","topic.creation.enable":"true","key.converter.schemas.enable":"true","database":"my_database","connection.uri":"mongodb://my_host:27017/my_database?replicaSet=rs&ssl=false","name":"mongodb-source-connector-xz6j","topic.creation.default.replication.factor":"-1","value.converter.schemas.enable":"true","connection.url":"mongodb://my_host:27017/my_database?replicaSet=rs&ssl=false","post.processor.chain":"com.mongodb.kafka.connect.sink.processor.DocumentIdAdder","value.converter":"org.apache.kafka.connect.storage.StringConverter","key.converter":"org.apache.kafka.connect.storage.StringConverter"},"tasks":[{"connector":"mongodb-source-connector-xz6j","task":0}],"type":"source"}}}

Can you please tell me where I made a mistake?

JIRA Link: K8S-406

[weeco]

Hey @arthurfedorov , could you show us the Kafka connect config in Console? I think I've seen a similar case where a timeout of 0 was used which meant the request immediately times out as soon as it was sent.

[arthurfedorov]

Hello @weeco, thank you for your answer I don't really understand what configs I need to show, can you tell me exactly what I need? I went through the values.yaml helm chart and didn't find any mention of timeouts

[weeco]

You can also describe the console pod and provide a copy of your mounted console config. It's a YAML file and the path to that file is either provided as an env variable or command line flag. Do not forget to redact sensitive configs

[arthurfedorov]

Config and describe fairly basic

k describe pod -n billing redpanda-console-58b444c78c-2j4lk
Name:             redpanda-console-58b444c78c-2j4lk
Namespace:        billing
Priority:         0
Service Account:  redpanda-console
Node:             cl1ilufmcc6ui3t4diop-udec/10.30.10.37
Start Time:       Wed, 16 Oct 2024 18:28:51 +0300
Labels:           app.kubernetes.io/instance=redpanda
                  app.kubernetes.io/name=console
                  pod-template-hash=58b444c78c
Annotations:      checksum-redpanda-chart/config: 8f92a2ce2476078fd80fbe21cd990fe140df1ea781ee463d6f17e2ee284ae7aa
                  checksum/config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
                  cni.projectcalico.org/containerID: 363959ded925e068cffd44e742bc0582b5308c01a57d0260d95c95b03138ece8
                  cni.projectcalico.org/podIP: 10.112.157.109/32
                  cni.projectcalico.org/podIPs: 10.112.157.109/32
Status:           Running
IP:               10.112.157.109
IPs:
  IP:           10.112.157.109
Controlled By:  ReplicaSet/redpanda-console-58b444c78c
Containers:
  console:
    Container ID:  containerd://82aea047b4f29a8169c8a2ccf8d4741eb6ea662952cfa865e3bf7e8c742bd93d
    Image:         docker.redpanda.com/redpandadata/console:v2.4.6
    Image ID:      docker.redpanda.com/redpandadata/console@sha256:e4cb11f01eebdf013a638353d962c86f75b5046dfc427539542ed768d7104499
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      --config.filepath=/etc/console/configs/config.yaml
    State:          Running
      Started:      Wed, 16 Oct 2024 18:37:27 +0300
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:http/admin/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:http/admin/health delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /etc/console/configs from configs (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwssq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  configs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redpanda-console
    Optional:  false
  kube-api-access-gwssq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

/app $ cat /etc/console/configs/config.yaml
# from .Values.console.config
connect:
  clusters:
  - name: connectors
    password: ""
    tls:
      caFilepath: ""
      certFilepath: ""
      enabled: false
      insecureSkipTlsVerify: false
      keyFilepath: ""
    token: ""
    url: http://redpanda-connectors.billing.svc.cluster.local:8083
    username: ""
  connectTimeout: 0
  enabled: true
  readTimeout: 0
  requestTimeout: 0
kafka:
  brokers:
  - redpanda-0.redpanda.billing.svc.cluster.local.:9093
  - redpanda-1.redpanda.billing.svc.cluster.local.:9093
  - redpanda-2.redpanda.billing.svc.cluster.local.:9093
  sasl:
    enabled: false
  schemaRegistry:
    enabled: true
    tls:
      caFilepath: ""
      certFilepath: ""
      enabled: false
      insecureSkipTlsVerify: false
      keyFilepath: ""
    urls:
    - http://redpanda-0.redpanda.billing.svc.cluster.local.:8081
    - http://redpanda-1.redpanda.billing.svc.cluster.local.:8081
    - http://redpanda-2.redpanda.billing.svc.cluster.local.:8081
  tls:
    caFilepath: ""
    certFilepath: ""
    enabled: false
    insecureSkipTlsVerify: false
    keyFilepath: ""
redpanda:
  adminApi:
    enabled: true
    tls:
      caFilepath: ""
      certFilepath: ""
      enabled: false
      insecureSkipTlsVerify: false
      keyFilepath: ""
    urls:
    - http://redpanda.billing.svc.cluster.local.:9644

[weeco]

connectTimeout: 0 is the problem. See my responses above. Please raise an issue in the helm repository to clarify why the defaults are faulty: https://github.com/redpanda-data/helm-charts

[david-yu]

Although commented out our docs currently provide an example of the following properties set: https://docs.redpanda.com/current/console/config/configure-console/

  # Used to test cluster connectivity.
  #connectTimeout: 15s
  # Overall REST timeout.
  #readTimeout: 60s
  # Timeout for REST requests.
  #requestTimeout: 6s

This was also uncovered through by a user in our community slack channel: https://redpandacommunity.slack.com/archives/C01AJDUT88N/p1730246418007929

[chrisseto]

Oh interesting, this looks like it's coming from the way that redpanda subcharts console (which directly imports the Console's config types)

@weeco could you give me a bit of education about how console's config loading works?

I'm guessing that koanf.UnmarshalConf is doing things much differently from json.Unmarshal. From what I can garner, the distinction between present and not present isn't encoded into the struct itself and instead is handled by koanf.UnmarshalConf.

We expected it to follow standard go unmarshalling behaviors 😅... The right answer might be running genpartial against the console types (which can also help with the yaml vs json tag issue). WDYT @RafalKorepta ?

[RafalKorepta]

The right answer might be running genpartial against the console types (which can also help with the yaml vs json tag issue). WDYT @RafalKorepta ?

Agree with that solution. That should work with any external function call.