search

kafka-ops / julie

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!
MIT License
417 stars 113 forks source link

Quotas not working out the box #562

Closed sionsmith closed 1 year ago

sionsmith commented 1 year ago

Describe the bug Reading the docs I should be able set quotas using the YAML examples provided in the documentation:

platform:
   kafka:
     quotas:
       - principal: "User:App0"
         producer_byte_rate: 1024
         consumer_byte_rate: 1024
         request_percentage: 50.0
       - principal: "User:App1"
         producer_byte_rate: 2048
         consumer_byte_rate: 2048
       - principal: "User:App2"
         request_percentage: 80.0

However quotas is not supported which throws the following error:

➜  julie git:(jenkins) ✗ java -jar julie-ops.jar --brokers localhost:29092 --clientConfig sasl_plain.properties --topology sasl-descriptor-topics-quotas.yaml --dryRun
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.admin.AdminClientConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
com.purbon.kafka.topology.exceptions.TopologyParsingException: Failed to deserialize topology from sasl-descriptor-topics-quotas.yaml
    at com.purbon.kafka.topology.serdes.TopologySerdes.deserialise(TopologySerdes.java:42)
    at com.purbon.kafka.topology.TopologyObjectBuilder.parseListOfTopologies(TopologyObjectBuilder.java:98)
    at com.purbon.kafka.topology.TopologyObjectBuilder.build(TopologyObjectBuilder.java:36)
    at com.purbon.kafka.topology.TopologyObjectBuilder.build(TopologyObjectBuilder.java:30)
    at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:127)
    at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:87)
    at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:211)
    at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:161)
    at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:147)
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "quotas" (class com.purbon.kafka.topology.model.users.platform.Kafka), not marked as ignorable (2 known properties: "rbac", "instances"])
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.purbon.kafka.topology.model.users.platform.Kafka["quotas"])
    at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:987)
    at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1974)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1701)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1679)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:330)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2798)
    at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3261)
    at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.parse(TopologyCustomDeserializer.java:177)
    at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.deserialize(TopologyCustomDeserializer.java:131)
    at com.purbon.kafka.topology.serdes.TopologyCustomDeserializer.deserialize(TopologyCustomDeserializer.java:47)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3413)
    at com.purbon.kafka.topology.serdes.TopologySerdes.deserialise(TopologySerdes.java:39)
    ... 8 more

To Reproduce java -jar julie-ops.jar --brokers localhost:29092 --clientConfig sasl_plain.properties --topology /julie/src/test/resource/descriptor-with-quotas.yaml

Expected behavior A quota would be added to the user

Runtime (please complete the following information):

Additional context Add any other context about the problem here.

sionsmith commented 1 year ago

I have just noticed that Quotas were not added until 4.4.0 my bad!