search

datavane / datavines

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.
https://datavane.github.io/datavines-website/
Apache License 2.0
430 stars 143 forks source link

[Bug] [Server] Error in configuring multiple metric of the same type for a Job #365

Closed xxzuo closed 6 months ago

xxzuo commented 6 months ago

Search before asking

What happened

Error in configuring multiple metric of the same type for a Job. 比如为 一个 Job 配置 两个 table_row_count 表行数检查, 任务无法执行

DataVines Version

dev

DataVines Config

-

Running Command

{
    "name": "test_1708612041714",
    "env": {
        "engine": "local",
        "type": "batch"
    },
    "sources": [
        {
            "plugin": "jdbc",
            "type": "source",
            "config": {
                "src_connector_type": "mysql",
                "password": "******",
                "database": "datavines",
                "driver": "com.mysql.cj.jdbc.Driver",
                "user": "root",
                "output_table": "dv_config",
                "table": "`datavines`.`dv_config`",
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"
            }
        },
        {
            "plugin": "jdbc",
            "type": "metadata",
            "config": {
                "src_connector_type": "mysql",
                "password": "******",
                "driver": "com.mysql.cj.jdbc.Driver",
                "user": "root",
                "output_table": "dv_actual_values",
                "table": "dv_actual_values",
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"
            }
        },
        {
            "plugin": "jdbc",
            "type": "source",
            "config": {
                "src_connector_type": "mysql",
                "password": "******",
                "database": "datavines",
                "driver": "com.mysql.cj.jdbc.Driver",
                "user": "root",
                "output_table": "`datavines`.`dv_config`",
                "table": "`datavines`.``datavines`.`dv_config``",
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"
            }
        }
    ],
    "transforms": [
        {
            "plugin": "sql",
            "type": "actual_value",
            "config": {
                "index": 0,
                "output_table": "invalidate_count_0db4a25f",
                "sql": "select count(1) as actual_value_0db4a25f from `datavines`.``datavines`.`dv_config``"
            }
        },
        {
            "plugin": "sql",
            "type": "expected_value_from_metadata_source",
            "config": {
                "index": 0,
                "output_table": "daily_range_0db4a25f",
                "sql": "select round(avg(actual_value),2) as expected_value_0db4a25f from dv_actual_values where data_time >=date_format('2024-02-22 22:27:21','%Y-%m-%d') and data_time < date_add(date_format('2024-02-22 22:27:21','%Y-%m-%d'), interval 1 DAY) and unique_code = 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y='"
            }
        },
        {
            "plugin": "sql",
            "type": "actual_value",
            "config": {
                "index": 0,
                "output_table": "invalidate_count_0db4a25f",
                "sql": "select count(1) as actual_value_0db4a25f from `datavines`.``datavines`.`dv_config``"
            }
        }
    ],
    "sinks": [
        {
            "plugin": "mysql",
            "type": "validate_result",
            "config": {
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai",
                "sql": "INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (null, '0db4a25f', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_0db4a25f}, ${expected_value_0db4a25f}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:27:21', '2024-02-22 22:27:21') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_0db4a25f}',update_time='2024-02-22 22:27:21'",
                "src_connector_type": "mysql",
                "password": "******",
                "driver": "com.mysql.cj.jdbc.Driver",
                "metric_unique_key": "0db4a25f",
                "expected_value": "expected_value_0db4a25f",
                "user": "root",
                "output_table": "dv_job_execution_result",
                "table": "dv_job_execution_result"
            }
        },
        {
            "plugin": "mysql",
            "type": "actual_value",
            "config": {
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai",
                "sql": "INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (null, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_0db4a25f}, '2024-02-22 22:27:21', '2024-02-22 22:27:21', '2024-02-22 22:27:21')",
                "src_connector_type": "mysql",
                "password": "******",
                "driver": "com.mysql.cj.jdbc.Driver",
                "metric_unique_key": "0db4a25f",
                "expected_value": "expected_value_0db4a25f",
                "user": "root",
                "output_table": "dv_actual_values",
                "table": "dv_actual_values"
            }
        },
        {
            "plugin": "mysql",
            "type": "validate_result",
            "config": {
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai",
                "sql": "INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (null, '0db4a25f', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_0db4a25f}, ${expected_value_0db4a25f}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:27:21', '2024-02-22 22:27:21') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_0db4a25f}',update_time='2024-02-22 22:27:21'",
                "src_connector_type": "mysql",
                "password": "******",
                "driver": "com.mysql.cj.jdbc.Driver",
                "metric_unique_key": "0db4a25f",
                "expected_value": "expected_value_0db4a25f",
                "user": "root",
                "output_table": "dv_job_execution_result",
                "table": "dv_job_execution_result"
            }
        },
        {
            "plugin": "mysql",
            "type": "actual_value",
            "config": {
                "url": "jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai",
                "sql": "INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (null, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_0db4a25f}, '2024-02-22 22:27:21', '2024-02-22 22:27:21', '2024-02-22 22:27:21')",
                "src_connector_type": "mysql",
                "password": "******",
                "driver": "com.mysql.cj.jdbc.Driver",
                "metric_unique_key": "0db4a25f",
                "expected_value": "expected_value_0db4a25f",
                "user": "root",
                "output_table": "dv_actual_values",
                "table": "dv_actual_values"
            }
        }
    ]
}

Error Exception

[INFO] 2024-02-22 22:21:38.572 io.datavines.engine.core.config.ConfigParser:[61] - Loading config file: {"name":"test_1708611698113","env":{"engine":"local","type":"batch"},"sources":[{"plugin":"jdbc","type":"source","config":{"src_connector_type":"mysql","password":"123456","database":"datavines","driver":"com.mysql.cj.jdbc.Driver","post_sql":"","pre_sql":"","user":"root","output_table":"dv_config","table":"`datavines`.`dv_config`","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}},{"plugin":"jdbc","type":"metadata","config":{"src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","user":"root","output_table":"dv_actual_values","table":"dv_actual_values","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}},{"plugin":"jdbc","type":"source","config":{"src_connector_type":"mysql","password":"123456","database":"datavines","driver":"com.mysql.cj.jdbc.Driver","post_sql":"","pre_sql":"","user":"root","output_table":"`datavines`.`dv_config`","table":"`datavines`.``datavines`.`dv_config``","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}}],"transforms":[{"plugin":"sql","type":"actual_value","config":{"index":0,"output_table":"invalidate_count_5f74b085","sql":"select count(1) as actual_value_5f74b085 from `datavines`.``datavines`.`dv_config``"}},{"plugin":"sql","type":"expected_value_from_metadata_source","config":{"index":0,"output_table":"daily_range_5f74b085","sql":"select round(avg(actual_value),2) as expected_value_5f74b085 from dv_actual_values where data_time >=date_format('2024-02-22 22:21:38','%Y-%m-%d') and data_time < date_add(date_format('2024-02-22 22:21:38','%Y-%m-%d'), interval 1 DAY) and unique_code = 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y='"}},{"plugin":"sql","type":"actual_value","config":{"index":0,"output_table":"invalidate_count_5f74b085","sql":"select count(1) as actual_value_5f74b085 from `datavines`.``datavines`.`dv_config``"}}],"sinks":[{"plugin":"mysql","type":"validate_result","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (32, '5f74b085', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_5f74b085}, ${expected_value_5f74b085}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:21:38', '2024-02-22 22:21:38') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_5f74b085}',update_time='2024-02-22 22:21:38'","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_job_execution_result","table":"dv_job_execution_result"}},{"plugin":"mysql","type":"actual_value","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (32, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_5f74b085}, '2024-02-22 22:21:38', '2024-02-22 22:21:38', '2024-02-22 22:21:38')","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_actual_values","table":"dv_actual_values"}},{"plugin":"mysql","type":"validate_result","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (32, '5f74b085', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_5f74b085}, ${expected_value_5f74b085}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:21:38', '2024-02-22 22:21:38') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_5f74b085}',update_time='2024-02-22 22:21:38'","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_job_execution_result","table":"dv_job_execution_result"}},{"plugin":"mysql","type":"actual_value","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (32, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_5f74b085}, '2024-02-22 22:21:38', '2024-02-22 22:21:38', '2024-02-22 22:21:38')","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_actual_values","table":"dv_actual_values"}}]}
[INFO] 2024-02-22 22:21:38.573 io.datavines.engine.core.config.ConfigParser:[65] - config after parse: {"name":"test_1708611698113","env":{"engine":"local","type":"batch","config":null},"sources":[{"plugin":"jdbc","type":"source","config":{"src_connector_type":"mysql","password":"123456","database":"datavines","driver":"com.mysql.cj.jdbc.Driver","post_sql":"","pre_sql":"","user":"root","output_table":"dv_config","table":"`datavines`.`dv_config`","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}},{"plugin":"jdbc","type":"metadata","config":{"src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","user":"root","output_table":"dv_actual_values","table":"dv_actual_values","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}},{"plugin":"jdbc","type":"source","config":{"src_connector_type":"mysql","password":"123456","database":"datavines","driver":"com.mysql.cj.jdbc.Driver","post_sql":"","pre_sql":"","user":"root","output_table":"`datavines`.`dv_config`","table":"`datavines`.``datavines`.`dv_config``","url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai"}}],"transforms":[{"plugin":"sql","type":"actual_value","config":{"index":0,"output_table":"invalidate_count_5f74b085","sql":"select count(1) as actual_value_5f74b085 from `datavines`.``datavines`.`dv_config``"}},{"plugin":"sql","type":"expected_value_from_metadata_source","config":{"index":0,"output_table":"daily_range_5f74b085","sql":"select round(avg(actual_value),2) as expected_value_5f74b085 from dv_actual_values where data_time >=date_format('2024-02-22 22:21:38','%Y-%m-%d') and data_time < date_add(date_format('2024-02-22 22:21:38','%Y-%m-%d'), interval 1 DAY) and unique_code = 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y='"}},{"plugin":"sql","type":"actual_value","config":{"index":0,"output_table":"invalidate_count_5f74b085","sql":"select count(1) as actual_value_5f74b085 from `datavines`.``datavines`.`dv_config``"}}],"sinks":[{"plugin":"mysql","type":"validate_result","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (32, '5f74b085', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_5f74b085}, ${expected_value_5f74b085}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:21:38', '2024-02-22 22:21:38') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_5f74b085}',update_time='2024-02-22 22:21:38'","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_job_execution_result","table":"dv_job_execution_result"}},{"plugin":"mysql","type":"actual_value","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (32, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_5f74b085}, '2024-02-22 22:21:38', '2024-02-22 22:21:38', '2024-02-22 22:21:38')","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_actual_values","table":"dv_actual_values"}},{"plugin":"mysql","type":"validate_result","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_job_execution_result (`job_execution_id`, `metric_unique_key`, `metric_type`, `metric_name`, `metric_dimension`, `database_name`, `table_name`, `column_name`, `actual_value`, `expected_value`, `expected_type`, `result_formula`, `operator`, `threshold`, `create_time`, `update_time`) VALUES (32, '5f74b085', 'single_table', 'table_row_count', 'completeness', 'datavines', '`datavines`.`dv_config`', '``', ${actual_value_5f74b085}, ${expected_value_5f74b085}, 'none', 'count', 'eq', 0.0, '2024-02-22 22:21:38', '2024-02-22 22:21:38') ON DUPLICATE KEY UPDATE actual_value = '${actual_value_5f74b085}',update_time='2024-02-22 22:21:38'","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_job_execution_result","table":"dv_job_execution_result"}},{"plugin":"mysql","type":"actual_value","config":{"job_execution_id":32,"url":"jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai","sql":"INSERT INTO dv_actual_values (job_execution_id, metric_name, unique_code, actual_value, data_time, create_time, update_time) VALUES (32, 'table_row_count', 'CZY5PHPKD8UEESP2XHUPP/L+7QSXF+CPR1YEW6+KX8Y=', ${actual_value_5f74b085}, '2024-02-22 22:21:38', '2024-02-22 22:21:38', '2024-02-22 22:21:38')","src_connector_type":"mysql","password":"123456","driver":"com.mysql.cj.jdbc.Driver","metric_unique_key":"5f74b085","expected_value":"expected_value_5f74b085","user":"root","output_table":"dv_actual_values","table":"dv_actual_values"}}]}
[INFO] 2024-02-22 22:21:38.574 [JobExecutionUniqueId=JOB-20240222-client_local_28:[65] - get connection success : jdbc:mysql://127.0.0.1:3306/datavines?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=Asia/Shanghai[username=root]
[INFO] 2024-02-22 22:21:38.583 [JobExecutionUniqueId=JOB-20240222-client_local_32:[90] - transform sql is: select count(1) as actual_value_5f74b085 from `datavines`.``datavines`.`dv_config``, transform_type is : actual_value
[ERROR] 2024-02-22 22:21:38.587 [JobExecutionUniqueId=JOB-20240222-client_local_32:[112] - transform execute error: 
java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`.`dv_config``' at line 1
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:110)
    at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
[ERROR] 2024-02-22 22:21:38.587 [JobExecutionUniqueId=JOB-20240222-client_local_32:[160] - execute error
io.datavines.common.exception.DataVinesException: transform execute error: 
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:123)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`.`dv_config``' at line 1
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:110)
    at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    ... 13 common frames omitted
[INFO] 2024-02-22 22:21:38.588 io.datavines.engine.core.BaseDataVinesBootstrap:[125] - Fatal Error, 

[INFO] 2024-02-22 22:21:38.588 io.datavines.engine.core.BaseDataVinesBootstrap:[126] - Reason: transform execute error: 

[INFO] 2024-02-22 22:21:38.589 io.datavines.engine.core.BaseDataVinesBootstrap:[127] - Exception StackTrace: io.datavines.common.exception.DataVinesException: transform execute error: 
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:123)
    at io.datavines.engine.local.api.LocalExecution.lambda$execute$0(LocalExecution.java:128)
    at java.util.ArrayList.forEach(ArrayList.java:1257)
    at io.datavines.engine.local.api.LocalExecution.execute(LocalExecution.java:119)
    at io.datavines.engine.core.BaseDataVinesBootstrap.parseConfigAndExecute(BaseDataVinesBootstrap.java:79)
    at io.datavines.engine.core.BaseDataVinesBootstrap.execute(BaseDataVinesBootstrap.java:43)
    at io.datavines.engine.local.executor.LocalEngineExecutor.execute(LocalEngineExecutor.java:44)
    at io.datavines.server.dqc.executor.runner.JobRunner.run(JobRunner.java:66)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`.`dv_config``' at line 1
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1218)
    at com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:110)
    at com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
    at io.datavines.engine.local.transform.sql.ActualValueExecutor.execute(ActualValueExecutor.java:46)
    at io.datavines.engine.local.transform.sql.SqlTransform.process(SqlTransform.java:96)
    ... 13 more

Engine Type

local

Java Version

1.8

Screenshots

No response

Are you willing to submit PR?

xxzuo commented 6 months ago

在 io.datavines.engine.config.BaseJobConfigurationBuilder#getMetricUniqueKey 中， 生成每个 job 的 MetricUniqueKey 时，对同一类型的 Metric 生成的 key 一样的 

    protected String getMetricUniqueKey(BaseJobParameter parameter) {
        return DigestUtils.md5Hex(String.format("%s_%s_%s_%s_%s",
                parameter.getMetricType(),
                parameter.getMetricParameter().get(DATABASE),
                parameter.getMetricParameter().get(TABLE),
                parameter.getMetricParameter().get(COLUMN),
                jobExecutionInfo.getId())).substring(0,8);
    }

所以这里 就会有问题 

String table = connectorFactory.getDialect().getFullQualifiedTableName(
                                                        metricInputParameter.get(DATABASE),
                                                        metricInputParameter.get(SCHEMA),
                                                        metricInputParameter.get(TABLE), true);
...
metricInputParameter.put(TABLE, table);

考虑 创建Job 时，禁止创建同样类型的 规则 或者修改 这部分代码逻辑

zixi0825 commented 6 months ago

374 已经解决了