search

datavane / datavines

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.
https://datavane.github.io/datavines-website/
Apache License 2.0
501 stars 155 forks source link

[Bug] [Engine] ColumnBlank metric cannot be used for datetime data type in mysql8.0.34 data source #342

Closed xxzuo closed 7 months ago

xxzuo commented 9 months ago

Search before asking

What happened

When configuring ColumnBlank metric for datetime type fields, an error is reported. mysql8.0.34 当表字段为 datetime类型时 无法执行 columnBlank 质量规则 image

DataVines Version

datavines version : dev

target datasource version: mysql 8.0.34

DataVines Config

nothing

Running Command

nothing

Error Exception

INFO] 2024-01-23 23:43:15.518 [JobExecutionUniqueId=JOB-20240123-client_local_130:[90] - transform sql is: select count(1) as actual_value_a73af8bb from invalidate_items_a73af8bb, transform_type is : actual_value
[ERROR] 2024-01-23 23:43:15.519 [JobExecutionUniqueId=JOB-20240123-client_local_130:[112] - transform execute error: 
java.sql.SQLException: Incorrect DATETIME value: ''
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2831)
    at com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
    at com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2828)
    at com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
    at com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:291)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[ERROR] 2024-01-23 23:43:15.523 [JobExecutionUniqueId=JOB-20240123-client_local_130:[160] - execute error
io.datavines.common.exception.DataVinesException: transform execute error: 
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:123)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Incorrect DATETIME value: ''
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2831)
    at com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
    at com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2828)
    at com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
    at com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:291)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    ... 13 common frames omitted
[ERROR] 2024-01-23 23:43:15.525 [JobExecutionUniqueId=JOB-20240123-client_local_129:[262] - drop invalidate_items_a73af8bb view error
[INFO] 2024-01-23 23:43:15.525 io.datavines.engine.core.BaseDataVinesBootstrap:[125] - Fatal Error, 

[INFO] 2024-01-23 23:43:15.525 io.datavines.engine.core.BaseDataVinesBootstrap:[126] - Reason: transform execute error: 

[INFO] 2024-01-23 23:43:15.526 io.datavines.engine.core.BaseDataVinesBootstrap:[127] - Exception StackTrace: io.datavines.common.exception.DataVinesException: transform execute error: 
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:123)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Incorrect DATETIME value: ''
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2831)
    at com.alibaba.druid.filter.FilterAdapter.statement_executeQuery(FilterAdapter.java:2503)
    at com.alibaba.druid.filter.FilterEventAdapter.statement_executeQuery(FilterEventAdapter.java:302)
    at com.alibaba.druid.filter.FilterChainImpl.statement_executeQuery(FilterChainImpl.java:2828)
    at com.alibaba.druid.proxy.jdbc.StatementProxyImpl.executeQuery(StatementProxyImpl.java:221)
    at com.alibaba.druid.pool.DruidPooledStatement.executeQuery(DruidPooledStatement.java:291)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    ... 13 more


### Engine Type

local

### Java Version

1.8

### Screenshots

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!