search

StarRocks / starrocks-connector-for-kafka

Apache License 2.0
7 stars 12 forks source link

we can't use debezium jdbc sink is because hibernate doesn't work with StarRocks. debezium jdbc insert error #20

Open alberttwong opened 7 months ago

alberttwong commented 7 months ago

The reason why we can't use debezium jdbc sink is because hibernate doesn't work with StarRocks. If we fix hibernate, then we don't need our own sink (unless you want to use stream load). 

2024-04-23 15:29:34 org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to determine Dialect without JDBC metadata (please set 'javax.persistence.jdbc.url', 'hibernate.connection.url', or 'hibernate.dialect')
2024-04-23 15:29:34     at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:277)
2024-04-23 15:29:34     at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
2024-04-23 15:29:34     at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
2024-04-23 15:29:34     at org.hibernate.boot.model.relational.Database.<init>(Database.java:45)
2024-04-23 15:29:34     at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.getDatabase(InFlightMetadataCollectorImpl.java:231)
2024-04-23 15:29:34     at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:199)
2024-04-23 15:29:34     at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:169)
2024-04-23 15:29:34     at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.build(MetadataBuildingProcess.java:128)
2024-04-23 15:29:34     at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:451)
2024-04-23 15:29:34     at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:102)
2024-04-23 15:29:34     at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:910)
2024-04-23 15:29:34     at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:960)
2024-04-23 15:29:34     at io.debezium.connector.jdbc.JdbcSinkConnectorTask.start(JdbcSinkConnectorTask.java:75)
2024-04-23 15:29:34     at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:329)
2024-04-23 15:29:34     at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
2024-04-23 15:29:34     at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
2024-04-23 15:29:34     at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)
2024-04-23 15:29:34     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2024-04-23 15:29:34     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-04-23 15:29:34     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2024-04-23 15:29:34     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2024-04-23 15:29:34     at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-23 15:29:34 Caused by: org.hibernate.HibernateException: Unable to determine Dialect without JDBC metadata (please set 'javax.persistence.jdbc.url', 'hibernate.connection.url', or 'hibernate.dialect')
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.determineDialect(DialectFactoryImpl.java:190)
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.buildDialect(DialectFactoryImpl.java:86)
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentWithDefaults(JdbcEnvironmentInitiator.java:229)
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:381)
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
2024-04-23 15:29:34     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
2024-04-23 15:29:34     at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
2024-04-23 15:29:34     at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
2024-04-23 15:29:34     ... 21 more
alberttwong commented 7 months ago 
2024-04-23 15:13:16 2024-04-23 22:13:16,087 WARN   ||  HHH000342: Could not obtain connection to query metadata   [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator]
2024-04-23 15:13:16 java.lang.NullPointerException
2024-04-23 15:13:16     at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcIsolationDelegate.delegateWork(JdbcIsolationDelegate.java:116)
2024-04-23 15:13:16     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
2024-04-23 15:13:16     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
2024-04-23 15:13:16     at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
2024-04-23 15:13:16     at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
2024-04-23 15:13:16     at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
2024-04-23 15:13:16     at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
2024-04-23 15:13:16     at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
2024-04-23 15:13:16     at org.hibernate.boot.model.relational.Database.<init>(Database.java:45)
2024-04-23 15:13:16     at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.getDatabase(InFlightMetadataCollectorImpl.java:231)
2024-04-23 15:13:16     at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:199)
2024-04-23 15:13:16     at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:169)
2024-04-23 15:13:16     at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.build(MetadataBuildingProcess.java:128)
2024-04-23 15:13:16     at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:451)
2024-04-23 15:13:16     at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:102)
2024-04-23 15:13:16     at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:910)
2024-04-23 15:13:16     at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:960)
2024-04-23 15:13:16     at io.debezium.connector.jdbc.JdbcSinkConnectorTask.start(JdbcSinkConnectorTask.java:75)
2024-04-23 15:13:16     at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:329)
2024-04-23 15:13:16     at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
2024-04-23 15:13:16     at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
2024-04-23 15:13:16     at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)
2024-04-23 15:13:16     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2024-04-23 15:13:16     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-04-23 15:13:16     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2024-04-23 15:13:16     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2024-04-23 15:13:16     at java.base/java.lang.Thread.run(Thread.java:829)
alberttwong commented 7 months ago

https://debezium.zulipchat.com/#narrow/stream/348104-community-mysql/topic/Unable.20to.20determine.20Dialect.20without.20JDBC.20metadata/near/435077946

alberttwong commented 7 months ago

This is very helpful https://rmoff.net/2020/01/16/changing-the-logging-level-for-kafka-connect-dynamically/

carlosescura commented 2 months ago

@alberttwong is there any update on this?

Is there any plan to support Hibernate in StarRocks? Or is it better to focus on developing this connector so it can handle the StarRocks specific scenarios like generating PK tables with custom DDL specs (just to mention a cool feature)?

alberttwong commented 2 months ago

@alberttwong is there any update on this?

Is there any plan to support Hibernate in StarRocks? Or is it better to focus on developing this connector so it can handle the StarRocks specific scenarios like generating PK tables with custom DDL specs (just to mention a cool feature)?

I don't work at StarRocks anymore. I'd ask in slack.