quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.4k stars 2.57k forks source link

Sub-optimal YAML support for Kafka config #24832

Open cdmikechen opened 2 years ago

cdmikechen commented 2 years ago

Describe the bug

I use application.yaml to config quarkus, and I use kafka extension. My application.yaml like this:

mp:
  messaging:
    incoming:
      test:
        topic: xxxx
        "auto.offset.reset": earliest
        "schema.registry.url": http://xxxx
        "key.deserializer": io.confluent.kafka.serializers.KafkaAvroDeserializer
        "value.deserializer": io.confluent.kafka.serializers.KafkaAvroDeserializer

At present, it seems that quarkus cannot directly recognize kafka's config like log.category.

Expected behavior

I hope the configuration of Kafka can be similar to log.category.

quarkus:
  log:
    min-level: DEBUG
    category:
      "org.apache":
        level: DEBUG

Instead of having to write like this, the readability of this writing method is certainly not very good, especially when there are many configured variables.

mp:
  messaging:
    incoming:
      mqtt:
        topic: xxxx
        auto:
          offset:
            reset: earliest
        schema:
          registry:
            url: http://xxxx
        key:
          deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
        value:
          deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer

Actual behavior

When the quarkus service starts, kafka will have the following warning:

[org.apa.kaf.cli.con.ConsumerConfig] (Quarkus Main Thread) The configuration '"schema.registry.url"' was supplied but isn't a known config.

How to Reproduce?

No

Output of uname -a or ver

dev

Output of java -version

11.0.14

GraalVM version (if different from Java)

GraalVM CE 22.0.0.2

Quarkus version or git rev

2.7.5

Build tool (ie. output of mvnw --version or gradlew --version)

3.6.3

Additional information

No

quarkus-bot[bot] commented 2 years ago

/cc @alesj, @cescoffier, @ozangunalp

alesj commented 2 years ago

So the problem is yaml format or that warning?

cdmikechen commented 2 years ago

@alesj Both of these. Quarkus get config is "schema.registry.url", "auto.offset.reset", "key.deserializer", "value.deserializer", so that kafka cannot recognize these variables.

alesj commented 2 years ago

But the other, more yaml-ish format works for you?

cescoffier commented 2 years ago

I think that log category is a specific case in Quarkus. I had a long discussion about that with @ia3andy

cdmikechen commented 2 years ago

But the other, more yaml-ish format works for you?

@alesj Quarkus can read config like this:

mp:
  messaging:
    incoming:
      mqtt:
        topic: xxxx
        auto:
          offset:
            reset: earliest
        schema:
          registry:
            url: http://xxxx
        key:
          deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
        value:
          deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer

But I think the readability of this writing method is certainly not very good, especially when there are many configured variables.

cdmikechen commented 2 years ago

I think that log category is a specific case in Quarkus. I had a long discussion about that with @ia3andy

@cescoffier So do you mean it can be supported? If this method is allowed, I can try to submit a PR to try to solve it.