Having trouble setting up Schema Registry with External and Internal Listener - Confluent For Kubernetes

[prottoyghose]

I am trying to deploy Schema Registry with an external and internal listener. I am using the Confluent For Kubernetes operator to deploy it. I keep getting the below exception
org.apache.kafka.common.config.ConfigException: Listener 'EXTERNAL://0.0.0.0:8081' has an unsupported scheme EXTERNAL. Here is the kubernetes config section for the Schema Registry

listeners:
    external:
      externalAccess:
        loadBalancer:
          annotations:
            external-dns.alpha.kubernetes.io/hostname:
            service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
          domain: mydomain.com
          externalTrafficPolicy: Cluster
          prefix: my-prefix
        type: loadBalancer
      tls:
        enabled: true
        secretRef: my-external-certificate-secret
    internal:
      tls:
        enabled: true
        secretRef: my-internal-certificate-secret

Here is the Schema Registry config that is displayed by the Schema Registry POD when the pod starts up

[INFO] 2023-06-22 14:59:34,240 [main] io.confluent.kafka.schemaregistry.security.config.SecureSchemaRegistryConfig logAll - SecureSchemaRegistryConfig values:
        access.control.allow.headers =
        access.control.allow.methods =
        access.control.allow.origin =
        access.control.skip.options = true
        authentication.method = NONE
        authentication.realm =
        authentication.roles = [*]
        authentication.skip.paths = []
        avro.compatibility.level =
        compression.enable = true
        confluent.license = 
        confluent.schema.registry.auth.mechanism = SSL
        confluent.schema.registry.auth.ssl.principal.mapping.rules = RULE:.*CN[s]?=[s]?([a-zA-Z0-9.-]*)?.*/$1/
        confluent.schema.registry.authorizer.class = io.confluent.kafka.schemaregistry.security.authorizer.schemaregistryacl.SchemaRegistryAclAuthorizer
        confluent.topic.acl.super.users =
        connector.connection.limit = 0
        csrf.prevention.enable = false
        csrf.prevention.token.endpoint = /csrf
        csrf.prevention.token.expiration.minutes = 30
        csrf.prevention.token.max.entries = 10000
        debug = true
        dos.filter.delay.ms = 100
        dos.filter.enabled = false
        dos.filter.insert.headers = true
        dos.filter.ip.whitelist = []
        dos.filter.managed.attr = false
        dos.filter.max.idle.tracker.ms = 30000
        dos.filter.max.requests.ms = 30000
        dos.filter.max.requests.per.connection.per.sec = 25
        dos.filter.max.requests.per.sec = 25
        dos.filter.max.wait.ms = 50
        dos.filter.throttle.ms = 30000
        dos.filter.throttled.requests = 5
        host.name = schema-registry-pod-0.namespace.svc.cluster.local
        http2.enabled = true
        idle.timeout.ms = 30000
        inter.instance.headers.whitelist = []
        inter.instance.listener.name = INTERNAL
        inter.instance.protocol = https
        kafkagroup.heartbeat.interval.ms = 3000
        kafkagroup.rebalance.timeout.ms = 300000
        kafkagroup.session.timeout.ms = 10000
        kafkastore.bootstrap.servers = [SSL://my-broker.namespace.svc.cluster.local:9071]
        kafkastore.checkpoint.dir = /tmp
        kafkastore.checkpoint.version = 0
        kafkastore.connection.url =
        kafkastore.group.id =
        kafkastore.init.timeout.ms = 60000
        kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
        kafkastore.sasl.kerberos.min.time.before.relogin = 60000
        kafkastore.sasl.kerberos.service.name =
        kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
        kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
        kafkastore.sasl.mechanism = GSSAPI
        kafkastore.security.protocol = SSL
        kafkastore.ssl.cipher.suites =
        kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
        kafkastore.ssl.endpoint.identification.algorithm =
        kafkastore.ssl.key.password = [hidden]
        kafkastore.ssl.keymanager.algorithm = SunX509
        kafkastore.ssl.keystore.location = /mnt/sslcerts/keystore.p12
        kafkastore.ssl.keystore.password = [hidden]
        kafkastore.ssl.keystore.type = JKS
        kafkastore.ssl.protocol = TLS
        kafkastore.ssl.provider =
        kafkastore.ssl.trustmanager.algorithm = PKIX
        kafkastore.ssl.truststore.location = /mnt/sslcerts/truststore.p12
        kafkastore.ssl.truststore.password = [hidden]
        kafkastore.ssl.truststore.type = JKS
        kafkastore.timeout.ms = 500
        kafkastore.topic = _schemas
        kafkastore.topic.replication.factor = 3
        kafkastore.topic.skip.validation = false
        kafkastore.update.handlers = []
        kafkastore.write.max.retries = 5
        leader.connect.timeout.ms = 60000
        leader.election.delay = false
        leader.eligibility = true
        leader.read.timeout.ms = 60000
        listener.protocol.map = [EXTERNAL:https, INTERNAL:https]
        listeners = [EXTERNAL://0.0.0.0:8081, INTERNAL://0.0.0.0:9081]
        master.eligibility = null
        metadata.encoder.old.secret = null
        metadata.encoder.secret = null
        metadata.encoder.topic = _schema_encoders
        metric.reporters = []
        metrics.num.samples = 2
        metrics.sample.window.ms = 30000
        metrics.tag.map = []
        mode.mutability = true
        nosniff.prevention.enable = false
        port = 8081
        proxy.protocol.enabled = false
        reject.options.request = false
        request.logger.name = io.confluent.rest-utils.requests
        request.queue.capacity = 2147483647
        request.queue.capacity.growby = 64
        request.queue.capacity.init = 128
        resource.extension.class = [io.confluent.kafka.schemaregistry.security.SchemaRegistrySecurityResourceExtension]
        resource.extension.classes = []
        resource.static.locations = []
        response.http.headers.config =
        response.mediatype.default = application/vnd.schemaregistry.v1+json
        response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
        rest.servlet.initializor.classes = []
        schema.cache.expiry.secs = 300
        schema.cache.size = 1000
        schema.canonicalize.on.consume = []
        schema.compatibility.level = backward
        schema.linking.rbac.enable = false
        schema.providers = []
        schema.registry.inter.instance.protocol =
        schema.registry.resource.extension.class = []
        server.connection.limit = 0
        shutdown.graceful.ms = 1000
        ssl.cipher.suites = []
        ssl.client.auth = true
        ssl.client.authentication = REQUIRED
        ssl.enabled.protocols = [TLSv1.2]
        ssl.endpoint.identification.algorithm = null
        ssl.key.password = [hidden]
        ssl.keymanager.algorithm =
        ssl.keystore.location = /mnt/sslcerts/keystore.p12
        ssl.keystore.password = [hidden]
        ssl.keystore.reload = false
        ssl.keystore.type = JKS
        ssl.keystore.watch.location =
        ssl.protocol = TLS
        ssl.provider =
        ssl.trustmanager.algorithm =
        ssl.truststore.location = /mnt/sslcerts/truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS
        suppress.stack.trace.response = true
        thread.pool.max = 200
        thread.pool.min = 8
        websocket.path.prefix = /ws
        websocket.servlet.initializor.classes = []

All the communications are on SSL for intra broker communication as well external. Kafka Version: 3.1 Schema Registry Image: 7.4.0

[OneCricketeer]

You'll see here listener.protocol.map isn't a listed config

https://docs.confluent.io/platform/current/schema-registry/installation/config.html

Since you're not explicitly setting listeners/map in your config, I suggest raising this issue for the operator itself rather than against the registry.

[prottoyghose]

@OneCricketeer I was looking at this PR here and tried to follow that which has a config listener.protocol.map. But you are right, it is not mentioned in the documentation.