feat: Only allow incompatible cast expressions to run in comet if a config is enabled

andygrove commented 2 weeks ago

Which issue does this PR close?

Part of https://github.com/apache/datafusion-comet/issues/286

Rationale for this change

We have discovered numerous compatibility issues with the CAST expression, so we should fall back to Spark for any cast operations that we do not fully support to avoid any data corruption or incorrect results.

This PR adds a new spark.comet.cast.allowIncompatible config and implements the mechanism to only allow incompatible casts when this is enabled.

What changes are included in this PR?

Move code for checking for supported cast operations out of QueryPlanSerde and into a new CometCast object
Add checks for all supported casts
Improve CometCastSuite to check that plan ran in Comet
Moved the documentation generator from common to spark module and updated it to generate a table showing which cast operations are compatible, incompatible, or unsupported
Explicitly enable spark.comet.cast.allowIncompatible in some tests that depend on incompatible casts

How are these changes tested?

Existing tests in CometCastSuite

andygrove commented 2 weeks ago

@vaibhawvipul Please take a look since this makes some changes to your recent code

andygrove commented 2 weeks ago

I plan on waiting until https://github.com/apache/datafusion-comet/pull/346 and https://github.com/apache/datafusion-comet/pull/340 are merged before merging this one, to avoid causing them to rebase again

andygrove commented 2 weeks ago

It looks like #340 will take a little longer, so will emrge this now

apache / datafusion-comet