Inconsistent behavior in HashJoin Projections

[adragomir]

Describe the bug

We ran into problems with projections inside HashJoin.

Each schema in the join (left / right) has:

• a single struct column
• and the join column (reference to a get_field inside the first column)

The projection is [0, 2] - the struct column from left, and the struct column from right

The join column is not specified in the output. When trying to optimize the join and reverse the order, the projection is swapped as [2, 0], however there is no column with index 2 in the output, as the output contains only the 2 structs

To Reproduce

• Create two schemas with a single struct column (key, value)
• Join on the key
• request the two value fields

Expected behavior

The hash join optimization works, even when swapping the join order (and wrapping in a ProjectionExec)

Additional context

Reading the comment for HashJoinExec::projection it says The projection indices of the columns in the output schema of join, however

• inside the try_new it seems to be checked against the join schema
• and inside the with_projection it seems to be checked against the output schema
• It also seems to be treated as relative to the join schema inside the swap_join_projection function - as it uses the left and right schemas

I tried taking a stab at it, but it's unclear what the meaning of what is passed in projections is. For now, I am fixing it surgically when swapping the order - I am rewriting the projections to be relative to the output schema when wrapping the join with a ProjectionExec

[adragomir]

You can see my temporary possible fix here: https://github.com/hstack/arrow-datafusion/commit/a4ab67d9dde53c3c13c92eb7070029282f0e837d

[my-vegetable-has-exploded]

It seems a bug related to me， thanks for catching it. I would take a look later.