Add MemoryReservation to batch splitting in joins

[alamb]

Is your feature request related to a problem or challenge?

Follow on to https://github.com/apache/datafusion/pull/12969 and https://github.com/apache/datafusion/issues/12633

In https://github.com/apache/datafusion/issues/12633 @mhilton noted that joins sometimes generate giant record batches which causes issues. @alihan-synnada fixed this in https://github.com/apache/datafusion/pull/12969 but internally sometimes the joins still generate giant output batches.

As @mhilton says in https://github.com/apache/datafusion/pull/12969#issuecomment-2418862655

Unfortunately this doesn't address the actual problem with creating giant batches, which is they require a lot of memory and that memory isn't accounted for in any MemoryPool. Wiring a MemoryReservation into BatchSplitter would probably be enough to address this though.

Describe the solution you'd like

I would like the memory accounting to take into account the large output batch

Describe alternatives you've considered

Wiring a MemoryReservation into BatchSplitter would probably be enough to address

Additional context

No response

[jatin510]

can i work on this task @alamb ?

[alamb]

@jatin510 of course -- see the guide here https://datafusion.apache.org/contributor-guide/index.html#open-contribution-and-assigning-tickets !

[jatin510]

take