search

awslabs / aws-glue-schema-registry

AWS Glue Schema Registry Client library provides serializers / de-serializers for applications to integrate with AWS Glue Schema Registry Service. The library currently supports Avro, JSON and Protobuf data formats. See https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html to get started.
Apache License 2.0
131 stars 97 forks source link

Can you build a single jar for all converters (avro, JsonSchema etc) #231

Open aflinton opened 2 years ago

aflinton commented 2 years ago

At the moment large shaded jars are created for a given schema registry type (e.g. avro (schema-registry-kafkaconnect-converter) or jsonschema (jsonschema-kafkaconnect-converter)). Is it possible/ could it be possible just to have all the options in one shaded jar? Otherwise I am having unpack zips for upload to S3 for use with MSK, change the avro jar for the jsonschema jar etc.

aflinton commented 2 years ago

Obviously I can copy the various "original" jars into the zip but it would possibly be easier to include the various schematypes in a single large shaded jar as then a user knows all the dependencies will be there.

mohitpali commented 2 years ago

Thanks for creating this issue. It could certainly be useful to create a combined shaded jar. We would have to change the module hierarchy or do some maven magic here. Can I also understand more on why using two shaded jars together does not work for you ?

adamflintonEmis commented 2 years ago

Just size really. Each shaded jar is about 50MB & contains almost exactly the same thing with a few classes missing at the specific converter level. If I include the "original" jars from the other converters I get what I am looking for but I simply think that it would be helpful to have an all in one. The basic use case is that we can start with JSON with a schema until we've got what we need then empty / rebuild the queue and switch to avro for speed etc.

blacktooth commented 1 year ago

Please upvote if this feature is important. We are also open to accepting PR if many users feel this feature is important.