search

getindata / kafka-connect-iceberg-sink

Apache License 2.0
77 stars 28 forks source link

Zolk 20 hadoop catalog and dependency updates #9

Closed wobu closed 1 year ago

wobu commented 1 year ago

Description

Reworked Plugin packaging to include default hadoop resources. This allows to use HDFS as a target with HadoopCatalog. Also Updated Iceberg to 1.0.0, Spark to 3.3 runtime and dependency of Kafka Connect API to 3.2.3

Tested it with a Strimzi Kafka Connect installation Unit tests run successfully

PR Checklist
gliter commented 1 year ago

Hi, looks good. Sorry for late response I was on vacations.

gliter commented 1 year ago

@wobu build fails due to output path change.

 Error:  Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (unpack-assembly) on project kafka-connect-iceberg-sink: An Ant BuildException has occured: Error while expanding /home/runner/work/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink/target/kafka-connect-iceberg-sink-0.1.4-SNAPSHOT-plugin.zip
Error:  java.io.FileNotFoundException: /home/runner/work/kafka-connect-iceberg-sink/kafka-connect-iceberg-sink/target/kafka-connect-iceberg-sink-0.1.4-SNAPSHOT-plugin.zip (No such file or directory)

Please also check if paths in build actions are not needed to be change as well.

gliter commented 1 year ago

There are still some errors during build. Does it pass locally for you?

wobu commented 1 year ago

not anymore, i'll investigate it later and fix it

gliter commented 1 year ago

@wobu looks like some property was missing. Fix was just merged. https://github.com/getindata/kafka-connect-iceberg-sink/pull/10#issuecomment-1315227751

wobu commented 1 year ago

merged upstream develop branch, but tests still failing. will investigate later