search

tabular-io / iceberg-kafka-connect

Apache License 2.0
203 stars 46 forks source link

Data sink from kafka to iceberg table to minio s3 zone lost data #173

Open NhatDuy11 opened 9 months ago

NhatDuy11 commented 9 months ago

Hello, I have sink a topic in kafka to the MINIO S3 area, in Object Storage there are metadata and data files, I use other tools to query the iceberg table but they have no data inside the table.

I am using iceberg-kafka-connect-runtime-hive-0.6.5

This is the config I'm using:

image

Full stack trace:

image

The metadata file is under the s3 MINIO area:

image image

I use other tools to query the iceberg table but they have no data inside the table:

image

Thank you everyone for looking at my article, do you have any ideas on how to fix this error? Please let me know. Thank you very much !

danielcweeks commented 9 months ago

Unfortunately, it looks like there's not enough info here really understand what's going on. If you see new data and metadata files (e.g. they were created by KC and not prior to starting the sink) then we know that the data access portion is working. I would suspect that either the sink is unable to update hive (repointing the table metadata location) or the sink is stopping before the commit interval. I don't see any snapshots in the metadata file, so I don't believe any data has been committed.

I think you'll need to look more closely at the logs and see if you can find an error.

thanhnv0901 commented 9 months ago

Please try with configurations in json file "value.converter.schemas.enable": "true", "schemas.enable": "true",

NhatDuy11 commented 9 months ago

@thanhnv0901 Your idea is very good, and I have successfully fixed the error, thank you very much

Vào Th 5, 21 thg 12, 2023 lúc 11:26 thanhnv0901 @.***> đã viết:

Please try with configurations in json file "value.converter.schemas.enable": "true", "schemas.enable": "true",

— Reply to this email directly, view it on GitHub https://github.com/tabular-io/iceberg-kafka-connect/issues/173#issuecomment-1865457994, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVVOWUH2EAAH4PE5WXANOOLYKO27JAVCNFSM6AAAAABA4OARISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRVGQ2TOOJZGQ . You are receiving this because you authored the thread.Message ID: @.***>

rj-u-developer commented 9 months ago

Getting below error when migrating the data from MSK to Iceberg in upsert mode.

Equality field ids shouldn't be null or empty when creating equality-delete writer.