search

eventuate-foundation / eventuate-cdc

Other
73 stars 31 forks source link

Updated mysql driver (mysql:mysql-connector-java:8.0.21/com.mysql.cj.jdbc.Driver) makes fails on current mysql database (5.7.13) #99

Open dartartem opened 3 years ago

dartartem commented 3 years ago

script build-and-test-mysql.sh shown failed tests, than stuck:

> Task :eventuate-local-java-cdc-connector-polling:test

io.eventuate.local.polling.PollingCdcProcessorEventsTest > shouldReadNewEventsOnly FAILED
    io.eventuate.util.test.async.EventuallyException at Eventually.java:70
        Caused by: java.lang.AssertionError at Assert.java:86

io.eventuate.local.polling.PollingCdcProcessorEventsTest > shouldReadUnprocessedEventsAfterStartup FAILED
    io.eventuate.util.test.async.EventuallyException at Eventually.java:70
        Caused by: java.lang.AssertionError at Assert.java:86

io.eventuate.local.polling.PollingCdcProcessorEventsTest > testCdcProcessingStatusService FAILED
    io.eventuate.util.test.async.EventuallyException at Eventually.java:70
        Caused by: java.lang.AssertionError at Assert.java:86
11:16:21.034 [SpringContextShutdownHook] INFO  o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'
11:16:21.035 [Curator-Framework-0] INFO  o.a.c.f.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
11:16:21.068 [SpringContextShutdownHook] INFO  o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 1000 ms.

io.eventuate.local.polling.PollingCdcProcessorCustomDBEventsTest > shouldReadNewEventsOnly FAILED
    io.eventuate.util.test.async.EventuallyException at Eventually.java:70
        Caused by: java.lang.AssertionError at Assert.java:86

io.eventuate.local.polling.PollingCdcProcessorCustomDBEventsTest > shouldReadUnprocessedEventsAfterStartup FAILED
    io.eventuate.util.test.async.EventuallyException at Eventually.java:70
        Caused by: java.lang.AssertionError at Assert.java:86
11:16:49.509 [SpringContextShutdownHook] INFO  o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'
11:16:49.513 [Curator-Framework-0] INFO  o.a.c.f.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
11:16:49.542 [SpringContextShutdownHook] INFO  o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 1000 ms.
11:16:58.767 [SpringContextShutdownHook] INFO  o.s.s.c.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'
11:16:58.768 [Curator-Framework-0] INFO  o.a.c.f.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
11:16:58.784 [SpringContextShutdownHook] INFO  o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 1000 ms.

mysql-binlog tests passed, polling tests failed.

After testing via Idea I found reason.

Method that retrieves primary keys: https://github.com/eventuate-foundation/eventuate-cdc/blob/wip-db-id-gen/eventuate-local-java-cdc-connector-polling/src/main/java/io/eventuate/local/polling/PollingDao.java#L238 throws exception:

java.lang.RuntimeException: Table io.eventuate.local.common.SchemaAndTable@1ca1bb96[schema=eventuate,tableName=events] has more than one primary key
    at io.eventuate.local.polling.PollingDao.queryPrimaryKey(PollingDao.java:238)
    at io.eventuate.local.polling.PollingDao.lambda$getPrimaryKey$4(PollingDao.java:214)
    at io.eventuate.local.common.DaoUtils.handleConnectionLost(DaoUtils.java:22)
    at io.eventuate.local.polling.PollingDao.getPrimaryKey(PollingDao.java:212)
    at io.eventuate.local.polling.PollingDao.processEvents(PollingDao.java:119)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:474)
    at io.eventuate.local.polling.PollingDao.start(PollingDao.java:99)
    at java.lang.Thread.run(Thread.java:748)

Under debug I found that that result set contains 2 primary keys with same value: "event_id".

I did not find why it works like this.

I am going to test it with mysql 8 (I just realized that since it is polling tests it is not necessary to update binlog connector).

dartartem commented 3 years ago

Tests on mysql8 also fail.

For now made this obvious change, it fixes the issue: https://gist.github.com/dartartem/db9aca1d275e8dc489dbb596643c3286

It just ignores if there are several PKs with the same name.

All mysql tests pass.

cer commented 3 years ago

Presumably, this is failing getting the primary key of the 'events' table. At the time, getPrimaryKeys() is called, how many tables called 'events' exist in one or more databases. I'd need to look up how the catalog, and schema arguments map to MySQL but perhaps it's not doing what you think. For example, a quick google search found a couple of examples like this getPrimaryKeys("my_database", null, "my_table").

Googling 'MySQL JDBC getPrimaryKeys' I also found this MySQL

dartartem commented 3 years ago

According this explanation: https://stackoverflow.com/a/7944489 catalogue is a different thing for each database. So I propose to specify catalogue only for mysql, for other databases left it null.