apache / shardingsphere

Distributed SQL transaction & query engine for data sharding, scaling, encryption, and more - on any database.
Apache License 2.0
19.82k stars 6.71k forks source link

How can we solve a large number of queries that contain in, if the parameter after in is large #24257

Open liuheng1104 opened 1 year ago

liuheng1104 commented 1 year ago

ok.... When can we optimize 'in' query for example: db1 (table_1, table_3, table_5), db2(table_0,table_2,table_4). sharding key is 'id'.sharding role is 'id %6'。 We have 3 records : id = 1, id = 2, id =4 。 so the real sql is : select from db1.table_1 where id in (1 ,2,4) , select from db2.table_2 where id in (1 ,2,4), select * from db2.table_4 where id in (1 ,2,4). so much in,it is not necessary i think。if we do batchGetByids => in (.............), it will do in all tables, it is inefficient i think。thanks

strongduanmu commented 1 year ago

@liuheng1104 Can you give an example for your question?

liuheng1104 commented 1 year ago

Our business is divided into 256 tables. Use the partition key (userId) to query in. The parameters are 100, and the logical sql is select * from table where user Id in (100 parameters), actual sql: select from table 1 where user Id in (100 parameters), select from table 2 where user Id in (100 parameters),

Many parameters after in are invalid code scanning

strongduanmu commented 1 year ago

Yes, this is an optimization point, I will try to optimize this problem later.