databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

Can't find iceberg-kaka-connect-hive connector on confluent #221

Closed ArkaSarkar19 closed 8 months ago

ArkaSarkar19 commented 8 months ago

Hi Team

There are two distributions that are provided i.e.iceberg-kafka-connect-runtime and iceberg-kafka-connect-runtime-hive in the ./kafka-connect-runtime/build/distributions folder. But I can't find the hive connector available on confluent-hub, the only available one one is https://www.confluent.io/hub/tabular/iceberg-kafka-connect, this connector doesn't have the necessary dependencies which are needed for hive catalog. Can the hive connector be also added to confluent-hub so that it can be installed using confluent-hub install command directly ,instead of manually adding the .zip file.

tabmatfournier commented 8 months ago

Hello.

Hadoop/Hive have too many security vulnerabilities which are flagged by ConfluentHub's security system which ends up removing the jar, so we no longer publish it there. It's not a problem with the connector, it's a problem with Hive.

You can go to the releases page (https://github.com/tabular-io/iceberg-kafka-connect/releases) and download the -HIVE version and manually upload it as a plugin in Confluent. E.g. image

ArkaSarkar19 commented 8 months ago

Hi Thanks for providing this information.