[Kernel][Expressions] Make the default expression handler evaluation lazy

[allisonport-db]

Feature request

Which Delta project/connector is this regarding?

• [ ] Spark
• [ ] Standalone
• [ ] Flink
• [X] Kernel
• [ ] Other (fill in here)

Overview

The expression handler should short-circuit on logical operators. This requires making all expression evaluation lazy since evaluation needs to be on a per-row basis.

First step: Convert all comparators (e.g. =, > etc) to be lazy. Rough idea on how to do that is:

• Create a comparator based on the data type of inputs. Both inputs are expected to have the same inputs.

  interface VectorComparator {
  /*
  * Compare the entry at given `rowId` in `left` and `right` vectors and return the comparison
  * result (-1, 0, 1)
  int compare(ColumnVector left, ColumnVector right, int rowId)
  }

  • We need utility method that can create these comparator. Pseudo code looks like below

    
    DataType dataType = left.getDataType();

  if (dataType instance BooleanType) { return new VectorComparator() { int compare(ColumnVector left, ColumnVector right, int rowId) { boolean leftResult = left.getBoolean(rowId); boolean rightResult = right.getBoolean(rowId); return Boolean.compare(leftResult, rightResult); } } else if (dataType instance of IntegerType) { .... } .. more types... }

• Return a ColumnVector that wraps the above created comparator and returns result based on the comparator operator type

  return new ColumnVector() {
  public getDataType() { return BooleanType.BOOLEAN;}
  
  public getSize() { argResults.leftResult.getSize(); }
  
  public isNullAt(int rowId) {
     boolean isLeftNull = argResults.leftResult.isNullAt(rowId);
     boolean isRightNull = argResults.rightResult.isNullAt(rowId);
     return isLeftNull || isRightNull;
  }
  
  public getBoolean(int rowId) {
    // check both are null. if yes return true, otherwise..
  
   // `==` will change depending upon the operator we are evaluating.
    compartor.compare(leftResult, rightResult, rowId) == 0
  }
  }

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently.
• [ ] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• [x] No. I cannot contribute this feature at this time.

[zzl-7]

FYI , I am working on this right now per recommendation from https://github.com/delta-io/delta/pull/2830#issuecomment-2032311536 Will submit a PR in a few days Thanks :)

[zzl-7]

Hi @vkorukanti I added lazy evaluation support for comparator expressions https://github.com/delta-io/delta/pull/2853 I have some question on the behavior of null comparison, if you have time can you take a look Thank you