search

datavane / datavines

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.
https://datavane.github.io/datavines-website/
Apache License 2.0
410 stars 139 forks source link

[Bug] [Server] When the database is case sensitive, there may be errors in fetching metadata #407

Closed xxzuo closed 2 months ago

xxzuo commented 2 months ago

Search before asking

What happened

  1. Prepare a case sensitive database
  2. Create two tables in the database, named TEST_TABLE and test_table 
    
create table TEST_TA
(
TEST1    varchar(255) null,
VALUE    varchar(255) null
)
comment 'TEST';

create table test_ta ( TEST1 varchar(255) null, VALUE varchar(255) null ) comment 'test';

3. Get the metadata of the database

The error is

[ERROR] 2024-06-19 19:05:43.156 io.datavines.server.scheduler.metadata.CatalogMetaDataFetchTaskRunner:[47] - fetch metadata error: org.springframework.dao.DuplicateKeyException:

Error updating database. Cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'

The error may exist in io/datavines/server/repository/mapper/CatalogEntityRelMapper.java (best guess)

The error may involve io.datavines.server.repository.mapper.CatalogEntityRelMapper.insert-Inline

The error occurred while setting parameters

SQL: INSERT INTO dv_catalog_entity_rel ( entity1_uuid, entity2_uuid, type, update_by, update_time ) VALUES ( ?, ?, ?, ?, ? )

Cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'

; Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'; nested exception is java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un' at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:247) at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70) at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:91) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:441) at com.sun.proxy.$Proxy98.insert(Unknown Source) at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:272) at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:59) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy$PlainMethodInvoker.invoke(MybatisMapperProxy.java:148) at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:89) at com.sun.proxy.$Proxy120.insert(Unknown Source) at com.baomidou.mybatisplus.extension.service.IService.save(IService.java:63) at com.baomidou.mybatisplus.extension.service.IService$$FastClassBySpringCGLIB$$f8525d18.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) at io.datavines.server.repository.service.impl.CatalogEntityRelServiceImpl$$EnhancerBySpringCGLIB$$ba75f734.save() at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.executeFetchDatabase(CatalogMetaDataFetchExecutorImpl.java:438) at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.lambda$executeFetchDataSource$1(CatalogMetaDataFetchExecutorImpl.java:232) at java.util.ArrayList.forEach(ArrayList.java:1257) at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.executeFetchDataSource(CatalogMetaDataFetchExecutorImpl.java:231) at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.execute(CatalogMetaDataFetchExecutorImpl.java:95) at io.datavines.server.scheduler.metadata.CatalogMetaDataFetchTaskRunner.run(CatalogMetaDataFetchTaskRunner.java:43) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un' at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117) at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955) at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:372) at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64) at com.sun.proxy.$Proxy157.update(Unknown Source) at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49) at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:106) at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62) at com.sun.proxy.$Proxy156.update(Unknown Source) at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194) at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181) at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427) ... 20 common frames omitted


### DataVines Version

dev

### DataVines Config

```conf
nothing

Running Command

fetch metadata

Error Exception

[ERROR] 2024-06-19 19:05:43.156 io.datavines.server.scheduler.metadata.CatalogMetaDataFetchTaskRunner:[47] - fetch metadata error: 
org.springframework.dao.DuplicateKeyException: 
### Error updating database.  Cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'
### The error may exist in io/datavines/server/repository/mapper/CatalogEntityRelMapper.java (best guess)
### The error may involve io.datavines.server.repository.mapper.CatalogEntityRelMapper.insert-Inline
### The error occurred while setting parameters
### SQL: INSERT INTO dv_catalog_entity_rel  ( entity1_uuid, entity2_uuid, type, update_by, update_time )  VALUES  ( ?, ?, ?, ?, ? )
### Cause: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'
; Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'; nested exception is java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'
    at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:247)
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70)
    at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:91)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:441)
    at com.sun.proxy.$Proxy98.insert(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:272)
    at com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:59)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy$PlainMethodInvoker.invoke(MybatisMapperProxy.java:148)
    at com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:89)
    at com.sun.proxy.$Proxy120.insert(Unknown Source)
    at com.baomidou.mybatisplus.extension.service.IService.save(IService.java:63)
    at com.baomidou.mybatisplus.extension.service.IService$$FastClassBySpringCGLIB$$f8525d18.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
    at io.datavines.server.repository.service.impl.CatalogEntityRelServiceImpl$$EnhancerBySpringCGLIB$$ba75f734.save(<generated>)
    at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.executeFetchDatabase(CatalogMetaDataFetchExecutorImpl.java:438)
    at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.lambda$executeFetchDataSource$1(CatalogMetaDataFetchExecutorImpl.java:232)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.executeFetchDataSource(CatalogMetaDataFetchExecutorImpl.java:231)
    at io.datavines.server.scheduler.metadata.task.CatalogMetaDataFetchExecutorImpl.execute(CatalogMetaDataFetchExecutorImpl.java:95)
    at io.datavines.server.scheduler.metadata.CatalogMetaDataFetchTaskRunner.run(CatalogMetaDataFetchTaskRunner.java:43)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '661ca869-3310-4dbe-b3b1-3150a0b4600e-5522d1d8-d5c8-4616-826b-944' for key 'dv_catalog_entity_rel.dv_entity_rel_un'
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:955)
    at com.mysql.cj.jdbc.ClientPreparedStatement.execute(ClientPreparedStatement.java:372)
    at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
    at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
    at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
    at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
    at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64)
    at com.sun.proxy.$Proxy157.update(Unknown Source)
    at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
    at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
    at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
    at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
    at com.baomidou.mybatisplus.extension.plugins.MybatisPlusInterceptor.intercept(MybatisPlusInterceptor.java:106)
    at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:62)
    at com.sun.proxy.$Proxy156.update(Unknown Source)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194)
    at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181)
    at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427)
    ... 20 common frames omitted

Engine Type

No response

Java Version

No response

Screenshots

No response

Are you willing to submit PR?