apache / shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Apache License 2.0
19.64k stars 6.67k forks source link

ShardingSphere-Proxy migration verification cannot be executed #29690

Open 6westboy9 opened 6 months ago

6westboy9 commented 6 months ago

Bug Report

When I used ShardingSphere-Proxy for data migration, I found that the migration was normal, but the validation could not be performed, and the error message was as follows:

cat ${ShardingSphere-Proxy-HOME}/logs/stdout.log image

3e324a83970baed8939d81b77939a71

ShardingSphere-Proxy server configuration is as follows:

cat ${ShardingSphere-Proxy-HOME}/conf/server.yaml

mode:
 type: Cluster
 repository:
   type: ZooKeeper
   props:
     namespace: governance_ds
     server-lists: xxx:2181
     retryIntervalMilliseconds: 500
     timeToLiveSeconds: 60
     maxRetries: 3
     operationTimeoutMilliseconds: 500

authority:
 users:
   - user: root@%
     password: root
   - user: sharding
     password: sharding
 privilege:
   type: ALL_PERMITTED

Which version of ShardingSphere did you use?

apache-shardingsphere-5.4.1-shardingsphere-proxy-bin.tar.gz

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

ShardingSphere-Proxy

Expected behavior

Successfully executed verification

Actual behavior

Failed to perform verification

Reason analyze (If you can)

After reading the source code, I found that the type attribute in YamProxyConfiguration is Cluster when starting, but when creating ContextManager, the type attribute in the modeConfiguration parameter in the ContextManagerBuilderParameter is Standalone, indicating that I don't quite understand it

image

image

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

CREATE DATABASE windranger_emr;
USE windranger_emr;

REGISTER STORAGE UNIT IF NOT EXISTS ds_0 (
    URL="jdbc:mysql://xxx:3306/windranger_emr?characterEncoding=utf8&serverTimezone=GMT%2B8&allowMultiQueries=true",
    USER="xxx",
    PASSWORD="xxx",
    PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000")
);

CREATE SHARDING TABLE RULE IF NOT EXISTS pat_inhos_order_group (
    STORAGE_UNITS(ds_0),
    SHARDING_COLUMN=inhos_code,TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="4"))
);

CREATE TABLE IF NOT EXISTS pat_inhos_order_group (
   // ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
REGISTER MIGRATION SOURCE STORAGE UNIT windranger_emr_source (
    URL="jdbc:mysql://xxx:3306/windranger_emr?characterEncoding=utf8&serverTimezone=GMT%2B8&allowMultiQueries=true",
    USER="xxx",
    PASSWORD="xxx",
    PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000")
);
MIGRATE TABLE windranger_emr_source.pat_inhos_order_group INTO windranger_emr.pat_inhos_order_group;
SHOW MIGRATION LIST;
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+
| id                                         | tables                                      | job_item_count | active | create_time         | stop_time |
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+
| j0102p0000013efc3a9791b6876cc092ed25155f84 | windranger_emr_source.pat_inhos_order_group | 1              | true   | 2024-01-09 17:04:25 | NULL      |
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+

SHOW MIGRATION STATUS 'j0102p0000013efc3a9791b6876cc092ed25155f84';
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| item | data_source           | tables                                      | status                   | active | processed_records_count | inventory_finished_percentage | incremental_idle_seconds | error_message |
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| 0    | windranger_emr_source | windranger_emr_source.pat_inhos_order_group | EXECUTE_INCREMENTAL_TASK | true   | 28735078                | 100                           |                          |               |
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
CHECK MIGRATION 'j0102p0000013efc3a9791b6876cc092ed25155f84' BY TYPE (NAME='CRC32_MATCH');
SHOW MIGRATION CHECK STATUS 'j0102p0000013efc3a9791b6876cc092ed25155f84';
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+
| tables                                      | result | check_failed_tables | active | inventory_finished_percentage | inventory_remaining_seconds | incremental_idle_seconds | check_begin_time        | check_end_time | duration_seconds | algorithm_type | algorithm_props | error_message |
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+
| windranger_emr_source.pat_inhos_order_group |        |                     | true   | 0                             | 0                           |                          | 2024-01-10 10:53:42.762 |                | 0                | CRC32_MATCH    |                 |               |
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+

Example codes for reproduce this issue (such as a github link).

sandynz commented 1 week ago

Hi @6westboy9 , migration and check will create internal Standalone data source, that's ok. From SHOW MIGRATION CHECK STATUS, looks check job is still running, you could try to verify it on underlying MySQL show processlist.