[CH] Diff of grouping sets query

KevinyhZou commented 1 month ago

Backend

CH (ClickHouse)

Bug description

do the query

select '2024-08-26' as day, id,name from
(
select
id, name
from test_tbl2
group by id, name
grouping sets((id),(id,name))
) where name  = 'a124'
;

the valina result is

the gluten result is empty.

Spark version

Spark-3.3.x

Spark configurations

No response

System information

No response

Relevant logs

No response

KevinyhZou commented 1 month ago

CHNativeColumnarToRow
+- ^(2) ProjectExecTransformer [2024-08-26 AS day#0, id#12L, name#13]
   +- ^(2) HashAggregateTransformer(keys=[id#12L, name#13, spark_grouping_id#11L], functions=[], isStreamingAgg=false)
      +- ^(2) InputIteratorTransformer[id#12L, name#13, spark_grouping_id#11L]
         +- ColumnarExchange hashpartitioning(id#12L, name#13, spark_grouping_id#11L, 1), ENSURE_REQUIREMENTS, [plan_id=112], [shuffle_writer_type=hash], [OUTPUT] List(id:LongType, name:StringType, spark_grouping_id:LongType)
            +- ^(1) HashAggregateTransformer(keys=[id#12L, name#13, spark_grouping_id#11L], functions=[], isStreamingAgg=false)
               +- ^(1) FilterExecTransformer (isnotnull(name#13) AND (name#13 = a124))
                  +- ^(1) ExpandExecTransformer [[id#6L, null, 1], [id#6L, name#7, 0]], [id#12L, name#13, spark_grouping_id#11L]
                     +- ^(1) ProjectExecTransformer [id#6L, name#7]
                        +- ^(1) NativeFileScan parquet default.test_tbl2[id#6L,name#7,day#8] Batched: true, DataFilters: [], Format: Parquet, Location: CatalogFileIndex(1 paths)[hdfs://testcluster/user/hive/warehouse/test_tbl2], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint,name:string>

分别需要针对[id#6L, null, 1], [id#6L, name#7, 0]], [id#12L, name#13, spark_grouping_id#11L] 三组fields 进行expand，按照ExpandTransform 逻辑，首先对[id#6L, null, 1]执行expand，数据到下游后，被filter where name = 'a124'，无行数输出，导致ISource中误认为数据已经结束