Added --replace_result - Githubissues

mjonss commented 9 months ago

Also updated example test result with latest master...

mjonss commented 9 months ago

@Defined2014 There were so many warnings about this from #52, so I created this PR :)

dveeden commented 9 months ago

After checking out this branch and merging upstream/master into it and building it I get this:

$ ./mysql-tester 
ERRO[0000] run test [example] err: sql:SELECT 2 FROM NON_EXISTING_TABLE;: failed to run query 
"SELECT 2 FROM NON_EXISTING_TABLE;" 
 around line 9, 
we need(103):
SELECT 2 FROM NON_EXISTING_TABLE;
SELECT 3 FROM NON_EXISTING_TABLE;
Got one of the listed errors
SELECT
but got(103):
SELECT 2 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist

ERRO[0000] run test [extensions] err: sql:SELECT 1 FROM NON_EXISTING_TABLE;: run "SELECT 1 FROM NON_EXISTING_TABLE;" at line 2 err, we got 
SELECT 1 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

but read result err EOF 

ERRO[0000] 2 tests failed                               
ERRO[0000] run test [example] err: sql:SELECT 2 FROM NON_EXISTING_TABLE;: failed to run query 
"SELECT 2 FROM NON_EXISTING_TABLE;" 
 around line 9, 
we need(103):
SELECT 2 FROM NON_EXISTING_TABLE;
SELECT 3 FROM NON_EXISTING_TABLE;
Got one of the listed errors
SELECT
but got(103):
SELECT 2 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist

ERRO[0000] run test [extensions] err: sql:SELECT 1 FROM NON_EXISTING_TABLE;: run "SELECT 1 FROM NON_EXISTING_TABLE;" at line 2 err, we got 
SELECT 1 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

but read result err EOF

And if I record this is the result:

diff --git a/r/example.result b/r/example.result
index 180d110..baf883e 100644
--- a/r/example.result
+++ b/r/example.result
@@ -6,8 +6,9 @@ a       b
 SELECT 1 FROM NON_EXISTING_TABLE;
 Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 2 FROM NON_EXISTING_TABLE;
+Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 3 FROM NON_EXISTING_TABLE;
-Got one of the listed errors
+Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 4;
 4
 4
@@ -20,16 +21,17 @@ SELECT 6;
 1 SELECT;
 Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "1 SELECT;" 
 2 SELECT;
+Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "2 SELECT;" 
 3 SELECT;
-Got one of the listed errors
+Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "3 SELECT;" 
 explain analyze format='brief' select * from t;
 id     estRows actRows task    access object   execution info  operator info   memory  disk
-TableReader    10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, tot_proc:<num>, tot_wait:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan      <num> Bytes     N/A
-└─TableFullScan        10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>, scan_detail: {total_process_keys:<num>, total_process_keys_size:<num>, total_keys:<num>, get_snapshot_time:<num>, rocksdb: {key_skipped_count:<num>, block: {}}}   keep order:false, stats:pseudo  N/A     N/A
+TableReader    10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan      <num> Bytes     N/A
+└─TableFullScan        10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>}     keep order:false, stats:pseudo  N/A     N/A
 explain analyze select * from t;
 id     estRows actRows task    access object   execution info  operator info   memory  disk
-TableReader_5  10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, tot_proc:<num>, tot_wait:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan_4    <num> Bytes     N/A
-└─TableFullScan_4      10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>, scan_detail: {total_process_keys:<num>, total_process_keys_size:<num>, total_keys:<num>, get_snapshot_time:<num>, rocksdb: {key_skipped_count:<num>, block: {}}}   keep order:false, stats:pseudo  N/A     N/A
+TableReader_5  10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan_4    <num> Bytes     N/A
+└─TableFullScan_4      10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>}     keep order:false, stats:pseudo  N/A     N/A
 select "My Matched matched changed Changed"
 My Matched matched changed Changed
 My Changed matched changed Changed
diff --git a/r/extensions.result b/r/extensions.result
index 814e650..983b030 100644
--- a/r/extensions.result
+++ b/r/extensions.result
@@ -1,2 +1,2 @@
 SELECT 1 FROM NON_EXISTING_TABLE;
-Got one of the listed errors
+Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

This is with:

Release Version: v7.6.0-alpha-391-gbe62f754fb
Edition: Community
Git Commit Hash: be62f754fb4182a544b52b2f58a5f867136e7695
Git Branch: master
UTC Build Time: 2023-12-07 08:02:16
GoVersion: go1.21.0
Race Enabled: false
Check Table Before Drop: false
Store: unistore

Defined2014 commented 9 months ago

After checking out this branch and merging upstream/master into it and building it I get this:

$ ./mysql-tester 
ERRO[0000] run test [example] err: sql:SELECT 2 FROM NON_EXISTING_TABLE;: failed to run query 
"SELECT 2 FROM NON_EXISTING_TABLE;" 
 around line 9, 
we need(103):
SELECT 2 FROM NON_EXISTING_TABLE;
SELECT 3 FROM NON_EXISTING_TABLE;
Got one of the listed errors
SELECT
but got(103):
SELECT 2 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist

ERRO[0000] run test [extensions] err: sql:SELECT 1 FROM NON_EXISTING_TABLE;: run "SELECT 1 FROM NON_EXISTING_TABLE;" at line 2 err, we got 
SELECT 1 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

but read result err EOF 

ERRO[0000] 2 tests failed                               
ERRO[0000] run test [example] err: sql:SELECT 2 FROM NON_EXISTING_TABLE;: failed to run query 
"SELECT 2 FROM NON_EXISTING_TABLE;" 
 around line 9, 
we need(103):
SELECT 2 FROM NON_EXISTING_TABLE;
SELECT 3 FROM NON_EXISTING_TABLE;
Got one of the listed errors
SELECT
but got(103):
SELECT 2 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist

ERRO[0000] run test [extensions] err: sql:SELECT 1 FROM NON_EXISTING_TABLE;: run "SELECT 1 FROM NON_EXISTING_TABLE;" at line 2 err, we got 
SELECT 1 FROM NON_EXISTING_TABLE;
Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

but read result err EOF

And if I record this is the result:

diff --git a/r/example.result b/r/example.result
index 180d110..baf883e 100644
--- a/r/example.result
+++ b/r/example.result
@@ -6,8 +6,9 @@ a       b
 SELECT 1 FROM NON_EXISTING_TABLE;
 Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 2 FROM NON_EXISTING_TABLE;
+Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 3 FROM NON_EXISTING_TABLE;
-Got one of the listed errors
+Error 1146 (42S02): Table 'example.NON_EXISTING_TABLE' doesn't exist
 SELECT 4;
 4
 4
@@ -20,16 +21,17 @@ SELECT 6;
 1 SELECT;
 Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "1 SELECT;" 
 2 SELECT;
+Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "2 SELECT;" 
 3 SELECT;
-Got one of the listed errors
+Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "3 SELECT;" 
 explain analyze format='brief' select * from t;
 id     estRows actRows task    access object   execution info  operator info   memory  disk
-TableReader    10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, tot_proc:<num>, tot_wait:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan      <num> Bytes     N/A
-└─TableFullScan        10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>, scan_detail: {total_process_keys:<num>, total_process_keys_size:<num>, total_keys:<num>, get_snapshot_time:<num>, rocksdb: {key_skipped_count:<num>, block: {}}}   keep order:false, stats:pseudo  N/A     N/A
+TableReader    10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan      <num> Bytes     N/A
+└─TableFullScan        10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>}     keep order:false, stats:pseudo  N/A     N/A
 explain analyze select * from t;
 id     estRows actRows task    access object   execution info  operator info   memory  disk
-TableReader_5  10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, tot_proc:<num>, tot_wait:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan_4    <num> Bytes     N/A
-└─TableFullScan_4      10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>, scan_detail: {total_process_keys:<num>, total_process_keys_size:<num>, total_keys:<num>, get_snapshot_time:<num>, rocksdb: {key_skipped_count:<num>, block: {}}}   keep order:false, stats:pseudo  N/A     N/A
+TableReader_5  10000.00        5       root    NULL    time:<num>, loops:<num>, RU:<num>, cop_task: {num:<num>, max:<num>, proc_keys:<num>, rpc_num:<num>, rpc_time:<num>, copr_cache_hit_ratio:<num>, build_task_duration:<num>, max_distsql_concurrency:<num>}      data:TableFullScan_4    <num> Bytes     N/A
+└─TableFullScan_4      10000.00        5       cop[tikv]       table:t tikv_task:{time:<num>, loops:<num>}     keep order:false, stats:pseudo  N/A     N/A
 select "My Matched matched changed Changed"
 My Matched matched changed Changed
 My Changed matched changed Changed
diff --git a/r/extensions.result b/r/extensions.result
index 814e650..983b030 100644
--- a/r/extensions.result
+++ b/r/extensions.result
@@ -1,2 +1,2 @@
 SELECT 1 FROM NON_EXISTING_TABLE;
-Got one of the listed errors
+Error 1146 (42S02): Table 'extensions.NON_EXISTING_TABLE' doesn't exist

This is with:

Release Version: v7.6.0-alpha-391-gbe62f754fb
Edition: Community
Git Commit Hash: be62f754fb4182a544b52b2f58a5f867136e7695
Git Branch: master
UTC Build Time: 2023-12-07 08:02:16
GoVersion: go1.21.0
Race Enabled: false
Check Table Before Drop: false
Store: unistore

Should enable check-error by manual. See https://github.com/pingcap/mysql-tester/blob/c5cf84ad0eefc3d685e210e75c9ff4d65cee66f1/src/main.go#L64

dveeden commented 9 months ago

@Defined2014 make test sets -check-error. Would be good to get #115 merged so we automatically check this.

Update: Actually, this isn't the case:

Makefile has a # before the ./mysql-tester -check-error line.
115 isn't starting tidb-server

Still fine to merge this PR and #115, but to actually get this tested more work is needed. Maybe we can use TiDB Action from @Icemap for this.

pingcap / mysql-tester

Added --replace_result #119

115 isn't starting `tidb-server`

pingcap / mysql-tester

Added --replace_result #119

115 isn't starting tidb-server

115 isn't starting `tidb-server`