bug fix: metadata buffer resident in Spark driver may go out of range

openucx / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer

https://www.sparkucx.org/

BSD 3-Clause "New" or "Revised" License

44 stars 31 forks source link

Open endersuu opened 1 year ago

endersuu commented 1 year ago

suppose to use map-side partition number, but mistakenly use reduce-side partition number.

Also, I believe this fix will help this issue https://github.com/openucx/sparkucx/issues/30