search

confluentinc / kafka-connect-datagen

Connector that generates data for demos
Apache License 2.0
20 stars 87 forks source link

CC-7254: Remove AvroConverter plugin from Confluent Hub archive #31

Closed C0urante closed 5 years ago

C0urante commented 5 years ago

The kafka-connect-avro-converter dependency is added to the POM for this connector so that the AvroData class can be used to convert from Avro to the Connect record format, but the actual AvroConverter class (or any of its nested classes) is never used. However, the Connect framework will still sometimes pick up the AvroConverter present in the plugin archive for the datagen connector and try to use that as an actual converter plugin, even though most of the transitive dependencies for the kafka-connect-avro-converter dependency are excluded, which then causes classloading issues. The changes here are a bit of a hack: they alter the Confluent Hub archive generated during the connector build by removing the AvroConverter class (and all of its nested classes) from the kafka-connect-avro-converter JAR file, so that the converter is never picked up as a plugin by the Connect framework. We could consider publishing the AvroData class as a separate Maven artifact so that people could add it as a dependency to their project without risk of this happening, but in the meantime, here's a quick-and-dirty solution that allows the connector to work alongside the Avro converter when both are loaded using the plugin path mechanism in Connect.

wicknicks commented 5 years ago

@chrise do you recall what classloading issues you ran into because of this setup?

C0urante commented 5 years ago

@wicknicks the dependencies for the Avro converter are excluded from packaging with this connector. So if the AvroConverter class packaged with the connector gets picked up as a converter plugin by Connect, it'll fail since a bunch of classes it relies on are missing.

C0urante commented 5 years ago

Pending a green build, merging into master to address issues with CC-7254