getindata / kafka-connect-iceberg-sink

Apache License 2.0
76 stars 27 forks source link

Writing throughput problems #42

Closed ndrluis closed 1 year ago

ndrluis commented 1 year ago

Hello, I'm experiencing a throughput issue, but I'm not sure how to resolve it. Currently, I'm taking a snapshot of a table that has 88 million records, and on the Debezium side, I'm achieving around 3.5 MB/s. However, when it comes to the Iceberg connector, the write speed doesn't exceed 500 KB/s.

Are there any JMX metrics that can help me understand where the bottleneck is? Do you have any configuration suggestions to improve this issue?

My configuration is following what's stated in the project's README.