FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Deserializer-properties are not propagated in delegating calls for serde creation #269

Open DarrenBishop opened 3 years ago

DarrenBishop commented 3 years ago

The call-site

    (jsa/serde reg
               {:avro.schema-registry/url schema-registry-url
                :avro.schema-registry/client schema-registry-client}
               {:avro/schema schema
                :key? key?
                :avro/coercion-cache coercion-cache
                :read-only? read-only?})

and the call-target

(defn serde
  "Given a type and logical type registry, a schema registry config with
  either a client or a URL and an Avro topic descriptor, build and
  return a Serde instance."
  [type-registry
   {:keys [avro.schema-registry/client
           avro.schema-registry/url]
    :as   registry-config}
   {:keys [avro/schema
           avro/coercion-cache
           key?
           deserializer-properties
           read-only?]
    :as   topic-config}]

Later in the code there are checks against property presence/values that cannot be true as a result; for example:

(let [avro-data (if (get deserializer-properties "specific.avro.reader")
                                                    (.deserialize base-deserializer ^String topic #^bytes raw-data ^Schema avro-schema)
                                                    (.deserialize base-deserializer ^String topic #^bytes raw-data))]

I am new to Clojure and this library, so hope my assessment is sound (and useful).

cddr commented 3 years ago

I see what you mean. I think this would be a fairly easy fix if you want to have a shot at submitting a PR.

Having said that, if you're not using this feature yet, I'd give some thought to using io.confluent.kafka.streams.serdes.avro.GenericAvroSerde in combination with https://github.com/cddr/edn-avro. The readme of that project explains some of the mistakes I think we made with the avro serde in jackdaw and it attempts to solve the problem of marshalling data from clojure into avro and back in a way that is simpler, more interoperable with built-in Kafka tooling, and doesn't get in the way when you want to do more advanced things.