search

erdemcer / kafka-connect-oracle

Kafka Source Connector For Oracle
Apache License 2.0
349 stars 167 forks source link

Multi-thread for connector #108

Open dangokuson opened 3 years ago

dangokuson commented 3 years ago

Hi @erdemcer,

As my understood, connector is running as single thread. Therefore, it will be a problem if a connector will have to read log for a lot of tables. Is there anyway to increase number of thread for connector?

Thanks!

erdemcer commented 3 years ago

Hi, Single thread opens single logminer session and reads redo logs sequentially . Multiple threads cause multiple logminer session and overhead on database. Single thread is enough for reading. Did you test any case ? Thanks

dangokuson commented 3 years ago

Hi @erdemcer,

It is okay to create only one connector that will be responsible for reading the logs of 500 tables?

Best regards, Dang Son

On Thu, Oct 21, 2021 at 5:40 PM Erdem Çer @.***> wrote:

Hi, Single thread opens single logminer session and reads redo logs sequentially . Multiple threads cause multiple logminer session and overhead on database. Single thread is enough for reading. Did you test any case ? Thanks

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/erdemcer/kafka-connect-oracle/issues/108#issuecomment-948482762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUSHT6GYQXBSH5UVDKRER7LUH7U2RANCNFSM5GNDYYZQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

erdemcer commented 3 years ago

Hi, In my production i am using single connector to read more than 1500 tables . I have not seen any lag or gap because of performance issues. Of course test on your environment will give the best answer. Regards

dangokuson commented 2 years ago

@erdemcer I used a single connector to read 1000 tables, those tables are located on a different schema. However, the performance was not good then I separated it into several connectors, each connector will responsible for the different schema. After changing, the latency was still big, it was around 10 minutes for that message available on Kafka. I guess it was because of in my case there are a lot of event messages that have been generated for a while. For example, for some tables, there are around 1 million messages have been generated with 5~10 minutes.

Is there any way to improve its performance, and do you think single connector is good enough in this case?