search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.71k stars 3.28k forks source link

[Bug] InPredicate.toThrift(InPredicate.java:272) Error #30860

Open Moredespicablethananyoneelse opened 9 months ago

Moredespicablethananyoneelse commented 9 months ago

Search before asking

Version

commit 48d7df205f7c57980a44e3f14e84d731e0e7cabb Author: zhiqiang seuhezhiqiang@163.com Date: Fri Dec 1 11:59:47 20

What's Wrong?

// in 报错 2024-02-05 09:03:28,556 WARN (mysql-nio-pool-51|182413) [StmtExecutor.executeByLegacy():816] execute Exception. stmt[223, f28daffb2b944084-9762ffbbbf099e34] java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:496) ~[guava-32.1.2-jre.jar:?] at org.apache.doris.analysis.InPredicate.toThrift(InPredicate.java:272) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.analysis.Expr.treeToThriftHelper(Expr.java:1054) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.analysis.Expr.treeToThrift(Expr.java:1029) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThriftHelper(PlanNode.java:628) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThriftHelper(PlanNode.java:659) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThrift(PlanNode.java:611) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanFragment.toThrift(PlanFragment.java:300) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator$FragmentExecParams.toTPipelineParams(Coordinator.java:3544) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator.sendPipelineCtx(Coordinator.java:829) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator.exec(Coordinator.java:640) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.sendResult(StmtExecutor.java:1492) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1462) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:645) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:734) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:460) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:430) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:240) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:160) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:187) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:240) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_392]

What You Expected?

is that a bug ? wait for it fixed

How to Reproduce?

here is the sqls contains "create table clause" and the select clause which throw error above:

table 01

CREATE TABLE RECURSIVE_DIM_BAI_02_number ( ID VARCHAR(20), A_INT INT, B_DOUBLE DOUBLE, C_VARCHAR_1 VARCHAR(25), C_VARCHAR_2 VARCHAR(25), D_NUMBER DECIMAL(10), re_number1 DECIMAL(2,1), re_number2 DECIMAL(3,2) ) ENGINE=OLAP DUPLICATE KEY( ID ) COMMENT "OLAP" DISTRIBUTED BY HASH( ID ) BUCKETS auto PROPERTIES ( "replication_num" = "3" )

table02

CREATE TABLE RECURSIVE_FACT_BAI_01_number ( ID VARCHAR(20), A_INT_1 INT, A_INT_2 INT, B_DATE_1 DATE, B_DATE_2 DATE, C_DOUBLE_1 DOUBLE, C_DOUBLE_2 DOUBLE, D_TIMESTAMP_1 DATETIME (6), D_TIMESTAMP_2 DATETIME (6), E_VARCHAR_1 VARCHAR(22), E_VARCHAR_2 VARCHAR(22), F_LONG_1 BIGINT, F_LONG_2 BIGINT, G_CHAR_1 CHAR(10), G_CHAR_2 CHAR(10), H_DATE_1 DATE, H_DATE_2 DATE, H_DATE_3 DATE, H_TIMESTAMP_4 DATETIME (6), H_TIMESTAMP_5 DATETIME (6), H_TIMESTAMP_6 DATETIME (6), I_INTERVAL_YEAR_1 varchar(1000), I_INTERVAL_YEAR_2 varchar(1000), J_INTERVAL_DAY_1 varchar(1000), J_INTERVAL_DAY_2 varchar(1000), k_number DECIMAL(10), re_number1 DECIMAL(2,1), re_number2 DECIMAL(3,2), re_int_1 int, re_int_2 int, re_int_3 int, re_int_4 int, re_long_1 BIGINT, re_long_2 BIGINT, re_long_3 BIGINT, re_long_4 BIGINT, re_double_1 double, re_double_2 double, re_double_3 double, re_double_4 double, re_double_5 double, re_double_6 double, re_double_7 double, re_double_8 double, re_number_1 DECIMAL(38,0), re_number_2 DECIMAL(38,0), re_number_3 DECIMAL(38,0), re_number_4 DECIMAL(38,0), re_number_5 DECIMAL(38,2), re_number_6 DECIMAL(38,1), re_number_7 DECIMAL(38,2), re_number_8 DECIMAL(38,3) ) ENGINE=OLAP DUPLICATE KEY( ID ) COMMENT "OLAP" DISTRIBUTED BY HASH( ID ) BUCKETS auto PROPERTIES ( "replication_num" = "3" )

table03 create table extreme_table_number ( ord_id bigint, a_int int, b_int int, c_int int, d_long bigint, e_long bigint, f_long bigint, g_double double, h_double double, j_double double, k_date date, l_date date, m_date date, n_timestamp DATETIME(6), o_timestamp DATETIME(6), p_timestamp DATETIME(6), q_interval varchar(1000), r_interval varchar(1000), s_interval varchar(1000), t_interval varchar(1000),
u_interval varchar(1000), v_interval varchar(1000), w_number double , x_number double, y_number decimal(38,25), z_number double, a_decima double ,
b_decima double, c_decima decimal(38,25), d_decima decimal(38,25), e_char char(1), f_char char(10),
g_char char(255), h_varchar varchar(1), i_varchar varchar(20), j_varchar varchar(500), n_date date, m_timestamp datetime(6), o_number DECIMAL(3), p_number DECIMAL(10,2), q_number DECIMAL(5,3), ex_int int, ex_long bigint, ex_double8 double, ex_double14 double, ex_double15 double, ex_number1 DECIMAL(10), ex_number2 DECIMAL(16), ex_number3 DECIMAL(5,3), ex_number4 DECIMAL(36,14), ex_number5 DECIMAL(20), ex_number6 DECIMAL(28), ext_number1 DECIMAL(2,1), ext_number2 DECIMAL(3,2)) ENGINE=OLAP DUPLICATE KEY( ord_id ) COMMENT "OLAP" DISTRIBUTED BY HASH( ord_id ) BUCKETS auto PROPERTIES ( "replication_num" = "3" )

table04 CREATE TABLE RECURSIVE_DIM_BAI_01_number ( ID VARCHAR(20) , A_CHAR_1 CHAR(10), A_CHAR_2 CHAR(10), A_number DECIMAL(10,0), re_number1 DECIMAL(3,2), re_int_1 int, re_int_2 int, re_int_3 int, re_int_4 int, re_long_1 BIGINT, re_long_2 BIGINT, re_long_3 BIGINT, re_long_4 BIGINT, re_double_1 double, re_double_2 double, re_double_3 double, re_double_4 double, re_double_5 double, re_double_6 double, re_double_7 double, re_double_8 double, re_number_1 DECIMAL(38,0), re_number_2 DECIMAL(38,0), re_number_3 DECIMAL(38,0), re_number_4 DECIMAL(38,0), re_number_5 DECIMAL(38,1), re_number_6 DECIMAL(38,1), re_number_7 DECIMAL(38,2), re_number_8 DECIMAL(38,3) ) ENGINE=OLAP DUPLICATE KEY( ID ) COMMENT "OLAP" DISTRIBUTED BY HASH( ID ) BUCKETS auto PROPERTIES ( "replication_num" = "3" )

table05,this may be replicated table , you may skip this clause CREATE TABLE RECURSIVE_FACT_BAI_01_number ( ID VARCHAR(20), A_INT_1 INT, A_INT_2 INT, B_DATE_1 DATE, B_DATE_2 DATE, C_DOUBLE_1 DOUBLE, C_DOUBLE_2 DOUBLE, D_TIMESTAMP_1 DATETIME (6), D_TIMESTAMP_2 DATETIME (6), E_VARCHAR_1 VARCHAR(22), E_VARCHAR_2 VARCHAR(22), F_LONG_1 BIGINT, F_LONG_2 BIGINT, G_CHAR_1 CHAR(10), G_CHAR_2 CHAR(10), H_DATE_1 DATE, H_DATE_2 DATE, H_DATE_3 DATE, H_TIMESTAMP_4 DATETIME (6), H_TIMESTAMP_5 DATETIME (6), H_TIMESTAMP_6 DATETIME (6), I_INTERVAL_YEAR_1 varchar(1000), I_INTERVAL_YEAR_2 varchar(1000), J_INTERVAL_DAY_1 varchar(1000), J_INTERVAL_DAY_2 varchar(1000), k_number DECIMAL(10), re_number1 DECIMAL(2,1), re_number2 DECIMAL(3,2), re_int_1 int, re_int_2 int, re_int_3 int, re_int_4 int, re_long_1 BIGINT, re_long_2 BIGINT, re_long_3 BIGINT, re_long_4 BIGINT, re_double_1 double, re_double_2 double, re_double_3 double, re_double_4 double, re_double_5 double, re_double_6 double, re_double_7 double, re_double_8 double, re_number_1 DECIMAL(38,0), re_number_2 DECIMAL(38,0), re_number_3 DECIMAL(38,0), re_number_4 DECIMAL(38,0), re_number_5 DECIMAL(38,2), re_number_6 DECIMAL(38,1), re_number_7 DECIMAL(38,2), re_number_8 DECIMAL(38,3) ) ENGINE=OLAP DUPLICATE KEY( ID ) COMMENT "OLAP" DISTRIBUTED BY HASH( ID ) BUCKETS auto PROPERTIES ( "replication_num" = "3" )

// the sql throw errors above

    SELECT b.re_number1 cc1,b.re_number2 cc2

from RECURSIVE_DIM_BAI_02_number b INNER join RECURSIVE_FACT_BAI_01_number c on d_number=k_number group by b.re_number1,b.re_number2 HAVING b.re_number1 is not null and b.re_number2 is not null and (max(b.re_number1))not in(SELECT ext_number1 from extreme_table_number inner join RECURSIVE_DIM_BAI_01_number on ord_id=a_number where not exists(SELECT min(re_number1) from RECURSIVE_FACT_BAI_01_number WHERE re_number2=ext_number2 group by re_number1,re_number2 HAVING count(re_number1)>9 ) and year(K_DATE)>4990
)

// the stack logged in the fe.log

2024-02-05 09:03:28,556 WARN (mysql-nio-pool-51|182413) [StmtExecutor.executeByLegacy():816] execute Exception. stmt[223, f28daffb2b944084-9762ffbbbf099e34] java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:496) ~[guava-32.1.2-jre.jar:?] at org.apache.doris.analysis.InPredicate.toThrift(InPredicate.java:272) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.analysis.Expr.treeToThriftHelper(Expr.java:1054) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.analysis.Expr.treeToThrift(Expr.java:1029) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThriftHelper(PlanNode.java:628) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThriftHelper(PlanNode.java:659) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanNode.treeToThrift(PlanNode.java:611) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.planner.PlanFragment.toThrift(PlanFragment.java:300) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator$FragmentExecParams.toTPipelineParams(Coordinator.java:3544) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator.sendPipelineCtx(Coordinator.java:829) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.Coordinator.exec(Coordinator.java:640) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.sendResult(StmtExecutor.java:1492) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1462) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:645) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:734) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:460) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:430) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:240) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:160) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:187) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:240) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_392]

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

zhangstar333 commented 9 months ago

seems old planner have thrown error. you could session variables : set experimental_enable_nereids_planner = true;

Moredespicablethananyoneelse commented 9 months ago

seems old planner have thrown error. you could session variables : set experimental_enable_nereids_planner = true;

hi： @zhangstar333

it seems nereids planner also throws an exception. it falls bact to old planner.

mysql> show variables like '%nereids%'; +-------------------------------------+-------+---------------+---------+ | Variable_name | Value | Default_Value | Changed | +-------------------------------------+-------+---------------+---------+ | disable_nereids_rules | | | 0 | | dump_nereids_memo | false | false | 0 | | enable_nereids_dml | false | false | 0 | | enable_nereids_timeout | true | true | 0 | | enable_nereids_trace | false | false | 0 | | experimental_enable_nereids_planner | true | - | - | | nereids_cbo_penalty_factor | 0.7 | 0.7 | 0 | | nereids_star_schema_support | true | true | 0 | | nereids_trace_event_mode | all | all | 0 | | plan_nereids_dump | false | false | 0 | | trace_nereids | false | false | 0 | +-------------------------------------+-------+---------------+---------+ 11 rows in set (0.00 sec)

mysql> SELECT b.re_number1 cc1,b.re_number2 cc2 -> from RECURSIVE_DIM_BAI_02_number b INNER join RECURSIVE_FACT_BAI_01_number c on d_number=k_number -> group by b.re_number1,b.re_number2 -> HAVING b.re_number1 is not null and b.re_number2 is not null -> and (max(b.re_number1))not in(SELECT ext_number1 -> from extreme_table_number inner join RECURSIVE_DIM_BAI_01_number on ord_id=a_number -> where not exists(SELECT min(re_number1) -> from RECURSIVE_FACT_BAI_01_number -> WHERE re_number2=ext_number2 -> group by re_number1,re_number2 -> HAVING count(re_number1)>9 ) -> and year(K_DATE)>4990 -> ); ERROR 1105 (HY000): IllegalStateException, msg: java.lang.IllegalStateException: null mysql>

zhangstar333 commented 9 months ago

my code commit it's nears to master branch: 18737bfd0e2d28353fa0dd7056e17b3985a93d4b I create table and execute the sql, it's no error with set experimental_enable_nereids_planner = true; you could set enable_fallback_to_original_planner = false, it's will not fallback to old planner when failed, so could see the error of nereids.

image