search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
826 stars 164 forks source link

abs returns incorrect value in some cases #666

Open andygrove opened 4 months ago

andygrove commented 4 months ago

Describe the bug

SQL

SELECT c28, abs(c28) AS x FROM test1 ORDER BY c28;

Spark Plan

AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(2) Sort [c28#228L ASC NULLS FIRST], true, 0
   +- AQEShuffleRead coalesced
      +- ShuffleQueryStage 0
         +- Exchange rangepartitioning(c28#228L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=24514]
            +- *(1) Project [c28#228L, abs(c28#228L) AS x#25310L]
               +- *(1) ColumnarToRow
                  +- FileScan parquet [c28#228L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c28:bigint>
+- == Initial Plan ==
   Sort [c28#228L ASC NULLS FIRST], true, 0
   +- Exchange rangepartitioning(c28#228L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=24500]
      +- Project [c28#228L, abs(c28#228L) AS x#25310L]
         +- FileScan parquet [c28#228L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c28:bigint>

Comet Plan

AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   *(2) Sort [c28#228L ASC NULLS FIRST], true, 0
   +- AQEShuffleRead coalesced
      +- ShuffleQueryStage 0
         +- Exchange rangepartitioning(c28#228L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=24579]
            +- *(1) ColumnarToRow
               +- CometProject [c28#228L, x#25316L], [c28#228L, abs(c28#228L) AS x#25316L]
                  +- CometScan parquet [c28#228L] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c28:bigint>
+- == Initial Plan ==
   Sort [c28#228L ASC NULLS FIRST], true, 0
   +- Exchange rangepartitioning(c28#228L ASC NULLS FIRST, 200), ENSURE_REQUIREMENTS, [plan_id=24559]
      +- CometProject [c28#228L, x#25316L], [c28#228L, abs(c28#228L) AS x#25316L]
         +- CometScan parquet [c28#228L] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/home/andy/git/apache/datafusion-comet/fuzz-testing/test1.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c28:bigint>

First difference at row 11: Spark: -8779944534083813778,8779944534083813778 Comet: -8779944534083813778,-8779944534083813778

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

andygrove commented 4 months ago

This is likely a duplicate of https://github.com/apache/datafusion-comet/issues/642

vaibhawvipul commented 4 months ago

working on it.

vaibhawvipul commented 4 months ago

I am able to reproduce the issue.

kazuyukitanimura commented 2 months ago

Native abs is disabled for now