Kafka input TLS authentication support

[LDTips]

Hi! Is it possible for TLS support to be also added for Kafka input? Currently it is only available in Kafka output configuration. I tried using the same config keys in the input as for output, but I believe they get ignored and I keep getting an error that I am not authenticated to access the topic I have specified in the config.

[karimra]

It should be supported, the docs need an update. Add in this PR: https://github.com/openconfig/gnmic/pull/63 You should be able to set it this way:

inputs:
  input1:
    type: kafka
    ## other fields
    tls:
      cert-file: /path/to/cert/file
      key-file: /path/to/key/file
      ca-file: /path/to/ca/file
      skip-verify: true|false

[LDTips]

Hmm, then I don't understand why my output does work, but input gives errors. Here are the config snippets: One of the gNMIc containers has:

outputs:
  output1:
    type: kafka
    name: "gNMIc-Telemetry"
    address: "<broker_addr>:<broker_port>"
    topic: Telemetry
    sasl:
    #   # SASL user name
    #   user:
    #   # SASL password
    #   password:
    #   # SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER are supported
    #   mechanism:
    #   # token url for OAUTHBEARER SASL mechanism
    #   token-url:
    tls:
      ca-file: /rpc/tls/ca.crt
      cert-file: /rpc/tls/client.crt
      key-file: /rpc/tls/client.key
      skip-verify: true
    max-retry: 2
    timeout: 2s
    recovery-wait-time: 5s
    format: event
    insert-key: false
    add-target:
    target-template:
    split-events: false
    msg-template:
    override-timestamps: false
    num-workers: 1
    debug: true
    buffer-size: 0
    compression-codec: gzip
    enable-metrics: false
    event-processors:

The other (adapter) has:

inputs:
  input1:
    type: kafka
    name: ""
    sasl:
      ## SASL user name
      # user:
      ## SASL password
      # password:
      ## SASL mechanism: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 and OAUTHBEARER are supported
      # mechanism:
      ## token url for OAUTHBEARER SASL mechanism
      # token-url:
    tls:
      ca-file: /rpc/tls/ca.crt
      cert-file: /rpc/tls/client.crt
      key-file: /rpc/tls/client.key
      skip-verify: true
    address: "<broker_addr>:<broker_port>"
    topics: Telemetry
    session-timeout: 10s
    heartbeat-interval: 3s
    recovery-wait-time: 2s
    version:
    format: event
    debug: true
    num-workers: 1
    event-processors: []
    outputs:
      - output1

Yet when I look at the adapter logs (debug), I have repeating messages:

2024/07/16 07:46:23.390838 /home/runner/work/gnmic/gnmic/pkg/inputs/kafka_input/kafka_input.go:149: [kafka_input] worker-0 failed to start consumer, topics="Telemetry", group="gnmic-consumers" : kafka server: The client is not authorized to access this group
2024/07/16 07:46:23.390873 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/sarama.go:129: [kafka_input] client/metadata fetching metadata for [Telemetry-IOSXR] from broker <broker_addr>:<broker_port>
2024/07/16 07:46:23.391267 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/sarama.go:129: [kafka_input] client/coordinator requesting coordinator for gnmic-consumers from <broker_addr>:<broker_port>
2024/07/16 07:46:23.391475 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1259: [kafka_input] client was not authorized to access group gnmic-consumers while attempting to find coordinator
2024/07/16 07:46:23.391485 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1198: [kafka_input] client/coordinator retrying after 250ms... (2 attempts remaining)
2024/07/16 07:46:23.641871 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/sarama.go:129: [kafka_input] client/coordinator requesting coordinator for gnmic-consumers from <broker_addr>:<broker_port>
2024/07/16 07:46:23.642431 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1259: [kafka_input] client was not authorized to access group gnmic-consumers while attempting to find coordinator
2024/07/16 07:46:23.642453 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1198: [kafka_input] client/coordinator retrying after 250ms... (1 attempts remaining)
2024/07/16 07:46:23.892886 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/sarama.go:129: [kafka_input] client/coordinator requesting coordinator for gnmic-consumers from <broker_addr>:<broker_port>
2024/07/16 07:46:23.893481 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1259: [kafka_input] client was not authorized to access group gnmic-consumers while attempting to find coordinator
2024/07/16 07:46:23.893501 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1198: [kafka_input] client/coordinator retrying after 250ms... (0 attempts remaining)
2024/07/16 07:46:24.143956 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/sarama.go:129: [kafka_input] client/coordinator requesting coordinator for gnmic-consumers from <broker_addr>:<broker_port>
2024/07/16 07:46:24.144567 /home/runner/go/pkg/mod/github.com/!i!b!m/sarama@v1.43.1/client.go:1259: [kafka_input] client was not authorized to access group gnmic-consumers while attempting to find coordinator

Any ideas why?

[LDTips]

Alright sorry I think I was tired when reporting this. The error is clear - I forgot to set the group permissions. Defining them fixed the issue. I didn't realise gNMIc creates a group for consumers, which I did not configure. Thank you for being my duck :)