search

substrait-io / substrait-java

Apache License 2.0
77 stars 72 forks source link

fix(spark): incorrect deriveRecordType() for Expand #316

Closed andrew-coleman closed 5 days ago

andrew-coleman commented 2 weeks ago

In the Expand relation, the record type was being calculated incorrectly, leading to errors when round-tripping to protobuf and back.

andrew-coleman commented 2 weeks ago

Incorporates the test code changes in draft PR https://github.com/substrait-io/substrait-java/pull/314

andrew-coleman commented 2 weeks ago

Yes, I was on the wrong track entirely with this. Turns out I needed to transpose the projections matrix that's defined in Spark before conversion to Substrait (and vice-versa).

vbarua commented 2 weeks ago

I think we need to tweak the spec as well as the protobuf and written spec don't fully agree on the output type: https://github.com/substrait-io/substrait/issues/714