Closed Wuerike closed 10 months ago
You will need to update iceberg.catalog.uri
, localhost won't work in this case AFAIK.
I've also tried http://rest:8181
before, but then I got a timeout error in the connector
Try giving the REST catalog a hostname
The error changed, it seems its failing in a further step now
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:618) at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:336) at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237) at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206) at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204) at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259) at
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181) at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at
java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.iceberg.exceptions.ServiceFailureException: Server error: NotFoundException: Location does not exist: s3://warehouse/default/events/metadata/00000-922efb19-660e-4b67-a7ba-af58697a1762.metadata.json at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:162) at
org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:109) at
org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:93) at
org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:176) at
org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:285) at
org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:219) at
org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:320) at
org.apache.iceberg.rest.RESTClient.get(RESTClient.java:96) at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:294) at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:310) at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99) at
org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:92) at
io.tabular.iceberg.connect.data.IcebergWriter.<init>(IcebergWriter.java:46) at
io.tabular.iceberg.connect.data.IcebergWriterFactory.createWriter(IcebergWriterFactory.java:35) at
io.tabular.iceberg.connect.channel.Worker.lambda$getWriterForTable$7(Worker.java:249) at
java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134) at
io.tabular.iceberg.connect.channel.Worker.getWriterForTable(Worker.java:249) at
io.tabular.iceberg.connect.channel.Worker.lambda$routeRecordStatically$4(Worker.java:203) at
java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390) at
io.tabular.iceberg.connect.channel.Worker.routeRecordStatically(Worker.java:201) at
io.tabular.iceberg.connect.channel.Worker.save(Worker.java:190) at
java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at
io.tabular.iceberg.connect.channel.Worker.save(Worker.java:177) at
io.tabular.iceberg.connect.IcebergSinkTask.put(IcebergSinkTask.java:119) at
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:587) ... 11 more
It looks like you may need to create the MinIO bucket, the sink won’t do that
You'll also want to set CATALOG_S3_PATH__STYLE__ACCESS
to true for the REST catalog container
I'll try your setup a little later today. BTW this would be a nice thing to contribute once it is working, if you're interested.
Yeah, sure, I was thinking the same, once it works I wanna share it
I configured the schema registry better and now I got:
Caused by: java.lang.UnsupportedOperationException: Cannot convert type: java.lang.String at io.tabular.iceberg.connect.data.RecordConverter.convert(RecordConverter.java:91) at io.tabular.iceberg.connect.data.IcebergWriter.write(IcebergWriter.java:56)
Maybe I'm still messing up with the converters
Now (I think) I'm sure its conversion related:
NoSuchTableException: Table does not exist
At this test this was my kafka-connect docker-compose service:
kafka-connect:
build:
context: .
dockerfile: Dockerfile.KafkaConnect
hostname: kafka-connect
container_name: kafka-connect
ports:
- 8083:8083
depends_on:
- kafka
- schemaregistry
environment:
CONNECT_BOOTSTRAP_SERVERS: kafka:9092
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: _connect_configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_STORAGE_TOPIC: _connect_offset
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: _connect_status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry:8085
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry:8085
CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect
CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components"
networks:
iceberg_net:
Full error:
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:230) at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:156) at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:522) at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:499) at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:335) at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237) at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206) at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204) at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259) at
org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181) at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at
java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: at
org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:326) at
org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:88) at
org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$4(WorkerSinkTask.java:522) at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:180) at
org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:214) ... 14 more Caused by: org.apache.kafka.common.errors.SerializationException: java.io.CharConversionException: Invalid UTF-32 character 0x17a2269 (above 0x0010ffff) at
char #1, byte #7) at
org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:66) at
org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:324) ... 18 more Caused by: java.io.CharConversionException: Invalid UTF-32 character 0x17a2269 (above 0x0010ffff) at
char #1, byte #7) at
com.fasterxml.jackson.core.io.UTF32Reader.reportInvalid(UTF32Reader.java:193) at
com.fasterxml.jackson.core.io.UTF32Reader.read(UTF32Reader.java:159) at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._loadMore(ReaderBasedJsonParser.java:276) at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2442) at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:698) at
com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) at
com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3090) at
org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:64) ... 19 more
Are the messages you are producing JSON strings with a schema?
I can try running it, I think the only missing piece for me is your Dockerfile for KC
Now (I think) I'm sure its conversion related:
- I got
NoSuchTableException: Table does not exist
- I've created the table
- I got conversion error at io.tabular.iceberg.connect.data.RecordConverter.convert
At this test this was my kafka-connect docker-compose service:
kafka-connect: build: context: . dockerfile: Dockerfile.KafkaConnect hostname: kafka-connect container_name: kafka-connect ports: - 8083:8083 depends_on: - kafka - schemaregistry environment: CONNECT_BOOTSTRAP_SERVERS: kafka:9092 CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: _connect_configs CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1 CONNECT_OFFSET_STORAGE_TOPIC: _connect_offset CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1 CONNECT_STATUS_STORAGE_TOPIC: _connect_status CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1 CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry:8085 CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schemaregistry:8085 CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_REST_ADVERTISED_HOST_NAME: kafka-connect CONNECT_PLUGIN_PATH: "/usr/share/java,/usr/share/confluent-hub-components" networks: iceberg_net:
Full error:
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:230) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:156) at org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:522) at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:499) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:335) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:237) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:206) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[] to Kafka Connect data failed due to serialization error: at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:326) at org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:88) at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$4(WorkerSinkTask.java:522) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:180) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:214) ... 14 more Caused by: org.apache.kafka.common.errors.SerializationException: java.io.CharConversionException: Invalid UTF-32 character 0x17a2269 (above 0x0010ffff) at char #1, byte #7) at org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:66) at org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:324) ... 18 more Caused by: java.io.CharConversionException: Invalid UTF-32 character 0x17a2269 (above 0x0010ffff) at char #1, byte #7) at com.fasterxml.jackson.core.io.UTF32Reader.reportInvalid(UTF32Reader.java:193) at com.fasterxml.jackson.core.io.UTF32Reader.read(UTF32Reader.java:159) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._loadMore(ReaderBasedJsonParser.java:276) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2442) at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:698) at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4703) at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3090) at org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:64) ... 19 more
i got this error too
To share my setup easily, I've created a repo with these info, I did my best to demonstrate the config I've been using
https://github.com/Wuerike/docker-iceberg-kafka-connect
If you guys manage to make it work let me know please, so far i haven't succeeded
Following along here as well, @Wuerike could you include your Producer as well in your repro example?
I was producing directly in the kafka-ui, as the connector fails at the first message I didn't build a producer yet
I think you need to specify key.converter=org.apache.kafka.connect.storage.StringConverter
this was causing the issue for me and adding this fixed it. the task now runs, but nothing is being written I just get these logs
my connector task:
{
"connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector",
"iceberg.catalog.s3.secret-access-key": "password",
"iceberg.catalog.s3.endpoint": "http://minio:9000",
"topics": "dummy.data",
"tasks.max": "1",
"iceberg.control.commit.interval-ms": "20000",
"iceberg.catalog.client.region": "eu-west-1",
"iceberg.catalog.uri": "http://rest:8181",
"iceberg.tables": "demo_schema.dummy_data",
"value.converter.schemas.enable": "false",
"iceberg.catalog.warehouse": "s3://demo-iceberg/",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"iceberg.catalog.type": "rest",
"schemas.enable": "false",
"iceberg.catalog.s3.access-key-id": "admin"
}
[2023-10-06 10:37:41,484] INFO Sending event of type: COMMIT_REQUEST (io.tabular.iceberg.connect.channel.Channel)
[2023-10-06 10:38:11,633] INFO Commit timeout reached (io.tabular.iceberg.connect.channel.CommitState)
[2023-10-06 10:38:11,686] INFO Sending event of type: COMMIT_COMPLETE (io.tabular.iceberg.connect.channel.Channel)
[2023-10-06 10:38:11,697] INFO Commit b99c810d-a950-422c-ad8f-74d959a31751 complete, committed to 0 table(s), vtts null (io.tabular.iceberg.connect.channel.Coordinator)
im sending these kind of messages:
{"block_hash":"f266944e-c8b5-42f4-a557-a1c04f1c325e","block_number":1993,"block_timestamp":1696588481}
sometimes it does give me (and writes to the table):
2023-10-06 10:43:12,079] INFO Sending event of type: COMMIT_TABLE (io.tabular.iceberg.connect.channel.Channel)
[2023-10-06 10:43:12,088] INFO Commit complete to table atlas.dummy_data, snapshot 8328104263542356348, commit ID 1faeac5d-5484-44de-9bb4-10cefb6b7c42, vtts null (io.tabular.iceberg.connect.channel.Coordinator)
[2023-10-06 10:43:12,097] INFO Sending event of type: COMMIT_COMPLETE (io.tabular.iceberg.connect.channel.Channel)
[2023-10-06 10:43:12,127] INFO Commit 1faeac5d-5484-44de-9bb4-10cefb6b7c42 complete, committed to 1 table(s), vtts null (io.tabular.iceberg.connect.channel.Coordinator)
I guess I don't quite understand how the commit process to iceberg works?
My connector config it's pretty similar to yours but it never works
@hendoxc I got to the same behaviour as you, I produce messages, I got no error but also nothing is committed to the table.
Did you solved it somehow?
@hendoxc I got to the same behaviour as you, I produce messages, I got no error but also nothing is committed to the table.
Did you solved it somehow?
Yes, looking at the files in minio
it seems the files are commited
to the table on 5 min intervals, so I guess I'm mis-understanding how "iceberg.control.commit.interval-ms": "20000"
works.
so everything seems to be all working for me. I also switched to using avro
and schema registry and things worked a lot better.
@hendoxc are you saying you're seeing commits every 5 min even though you set the commit interval to 20 sec?
I think the problem might be that the config property iceberg.control.commit.interval-ms
is for sink versions 0.5.0
and later. If using 0.4.x
the property is iceberg.control.commitIntervalMs
, so it might have been using the default.
I have the same behavior running 0.5.0
. I see COMMIT REQUEST
and COMMIT COMPLETE
messages in the logs, but no data is written out to my catalog until I close kafka-connect . I'll experiment with it today and see if it writes after 5 minutes.
Using iceberg.control.commitIntervalMs
finally make it work
Now I'll update my repo to share a demo
Thanks you all
Awesome, nice work!
I've updated my repo with an exemple of how to execute everything locally https://github.com/wuerike/kafka-iceberg-streaming
@bryanck how can we use this work to contribute here? Don't know if copying + pasting to a folder here or just mentioning it in the readme
A link on the README would be great, perhaps at the bottom we can have a section for resources. This repo will be archived at some point once the code is moved to Iceberg.
Can you help me run it locally? I don't have experience either with Iceberg or Kafka, so I probably doing something wrong.
Starting from the tabular-io docker compose of iceberg + spark, I've added the Kafka services but I could manage to make it work end-to-end
I created a dockerfile to extend kafka connect image and install this pluging
Dockerfile:
docker-compose.yaml:
Config:
Error: