search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

No Dialect mapping for JDBC type: -9 (source from oracle 11) #92

Open 0xhanh opened 5 years ago

0xhanh commented 5 years ago

I pull data from oracle11g, ~ source oracle; below my flume-ng config:

agentTest.channels = channelTest
agentTest.sources = sourceTest
agentTest.sinks = sinkTest

###########sql source#################

# For each Test of the sources, the type is defined

agentTest.sources.sourceTest.type = org.keedio.flume.source.SQLSource
#agentTest.sources.sourceTest.type = org.keedio.flume.source.SQLServerCustomDialect
agentTest.sources.sourceTest.hibernate.connection.url = jdbc:oracle:thin:@:1521:orcl

# Hibernate Database connection properties

agentTest.sources.sourceTest.hibernate.connection.user = ...
agentTest.sources.sourceTest.hibernate.connection.password = ...
agentTest.sources.sourceTest.hibernate.connection.autocommit = true
agentTest.sources.sourceTest.hibernate.dialect = org.hibernate.dialect.Oracle10gDialect
agentTest.sources.sourceTest.hibernate.connection.driver_class = oracle.jdbc.driver.OracleDriver
agentTest.sources.sourceTest.run.query.delay=10000
agentTest.sources.sourceTest.enclose.by.quotes = false

agentTest.sources.sourceTest.status.file.path = /usr/local/flume
agentTest.sources.sourceTest.status.file.name = agentTest.sqlSource.status

# Custom query
agentTest.sources.sourceTest.start.from = 0
agentTest.sources.sourceTest.custom.query = my sql select

agentTest.sources.sourceTest.batch.size = 1000
agentTest.sources.sourceTest.max.rows = 1000
agentTest.sources.sourceTest.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider
agentTest.sources.sourceTest.hibernate.c3p0.min_size=1
agentTest.sources.sourceTest.hibernate.c3p0.max_size=10
...

Running, errors belowed:

2019-08-21 16:52:58,153 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:141)] Polling sink runner starting 2019-08-21 16:52:58,183 (PollableSourceRunner-SQLSource-sourceTest) [ERROR - org.keedio.flume.source.HibernateHelper.executeQuery(HibernateHelper.java:128)] Exception thrown, resetting connection. org.hibernate.MappingException: No Dialect mapping for JDBC type: -9 at org.hibernate.dialect.TypeNames.get(TypeNames.java:87) at org.hibernate.dialect.TypeNames.get(TypeNames.java:118) at org.hibernate.dialect.Dialect.getHibernateTypeName(Dialect.java:653) at org.hibernate.loader.custom.JdbcResultMetadata.getHibernateType(JdbcResultMetadata.java:93) at org.hibernate.loader.custom.ScalarResultColumnProcessor.performDiscovery(ScalarResultColumnProcessor.java:62) at org.hibernate.loader.custom.CustomLoader.autoDiscoverTypes(CustomLoader.java:498) at org.hibernate.loader.Loader.getResultSet(Loader.java:2074) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1863) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1839) at org.hibernate.loader.Loader.doQuery(Loader.java:910) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:355) at org.hibernate.loader.Loader.doList(Loader.java:2554) at org.hibernate.loader.Loader.doList(Loader.java:2540) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2370) at org.hibernate.loader.Loader.list(Loader.java:2365) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:353) at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:1909) at org.hibernate.internal.AbstractSessionImpl.list(AbstractSessionImpl.java:311) at org.hibernate.internal.SQLQueryImpl.list(SQLQueryImpl.java:141) at org.keedio.flume.source.HibernateHelper.executeQuery(HibernateHelper.java:126) at org.keedio.flume.source.SQLSource.process(SQLSource.java:102) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133) at java.lang.Thread.run(Thread.java:748)

0xhanh commented 5 years ago

Solved by upgrading Hibernate to 5.3