Push projections into hash joins

[andygrove]

What is the problem the feature request solves?

DataFusion has an optimization where projections can be pushed down into hash joins. This implemented in the projection pushdown optimizer rule and was added in https://github.com/apache/datafusion/pull/9236.

We do not currently apply this rule, so each HashJoin will output all columns and then we have a projection to discard the unwanted columns. Here is an example plan from TPC-DS q3 showing two projections after the outermost join. Pushing the projection into the join may show some small performance improvement.

AggregateExec: mode=Partial, gby=[col_0@0 as col_0, col_3@3 as col_1, col_2@2 as col_2], aggr=[sum]
  ProjectionExec: expr=[col_0@0 as col_0, col_2@2 as col_1, col_1@4 as col_2, col_2@5 as col_3]
    ProjectionExec: expr=[col_0@3 as col_0, col_1@4 as col_1, col_2@5 as col_2, col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2]
      HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_1@1)]
        CopyExec [UnpackOrDeepCopy]
          ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Utf8]
        CopyExec [UnpackOrClone]
          ProjectionExec: expr=[col_1@1 as col_0, col_1@3 as col_1, col_2@4 as col_2]
            HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_0@0)]
              CopyExec [UnpackOrDeepCopy]
                ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32]
              CopyExec [UnpackOrClone]
                FilterExec: col_0@0 IS NOT NULL AND col_1@1 IS NOT NULL
                  ScanExec: source=[CometScan parquet  (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Decimal128(7, 2)]

Describe the potential solution

No response

Additional context

No response