apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.14k stars 416 forks source link

[VL] Array with null value will fallback #6625

Open Yohahaha opened 1 month ago

Yohahaha commented 1 month ago

Backend

VL (Velox)

Bug description

val aai = Literal.create(Seq(Seq(1, 2, 3), null, Seq(4, 5)),
      ArrayType(ArrayType(IntegerType, containsNull = false), containsNull = true))

16:32:17.693 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=0], due to: Native validation failed: 
Optional[Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1306 function:validate, thrown from file:SubstraitToVeloxExpr.cc line:400 function:literalsToArrayVector, reason:(0 vs. 0) there should be at least 1 value in list literal.].

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

"literal": {
                          "list": {
                            "values": [
                              {
                                "list": {
                                  "values": [
                                    {
                                      "i32": 1
                                    },
                                    {
                                      "i32": 2
                                    },
                                    {
                                      "i32": 3
                                    }
                                  ]
                                }
                              },
                              {
                                "null": {
                                  "list": {
                                    "type": {
                                      "i32": {
                                        "nullability": "NULLABILITY_REQUIRED"
                                      }
                                    },
                                    "nullability": "NULLABILITY_NULLABLE"
                                  }
                                }
                              },
                              {
                                "list": {
                                  "values": [
                                    {
                                      "i32": 4
                                    },
                                    {
                                      "i32": 5
                                    }
                                  ]
                                }
                              }
                            ]
                          }
                        }
vectorijk commented 1 month ago

I am looking into this bug if no one takes this

Yohahaha commented 1 month ago

I am looking into this bug if no one takes this

great! feel free to take and fix it.

Yohahaha commented 1 month ago

https://github.com/apache/incubator-gluten/blob/531e82a5bb7c93603661f4e0c5d98317d58cf109/cpp/velox/substrait/SubstraitToVeloxExpr.cc#L407-L417

https://github.com/apache/incubator-gluten/blob/531e82a5bb7c93603661f4e0c5d98317d58cf109/cpp/velox/substrait/SubstraitToVeloxExpr.cc#L419-L455

I have debug the case and found above two functions need change to make ArrayVector compute nulls.