Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It is redundant to store schema information in the shuffle files, as we already have them as part of the execution plan / query stages.
Describe the solution you'd like
Use IPC streams instead of file format.
Is your feature request related to a problem or challenge? Please describe what you are trying to do. It is redundant to store schema information in the shuffle files, as we already have them as part of the execution plan / query stages.
Describe the solution you'd like Use IPC streams instead of file format.
Describe alternatives you've considered
Additional context