search

NVIDIA / sparkucx

A high-performance, scalable and efficient ShuffleManager plugin for Apache Spark, utilizing UCX communication layer
https://www.sparkucx.org/
BSD 3-Clause "New" or "Revised" License
22 stars 14 forks source link

do not save worker address in executor address map #29

Closed jeynmann closed 1 year ago

jeynmann commented 1 year ago

Do not save workerAddress in executorAddresses. If the transport receives connection requests from other executors and save the worker address in executorAddresses while preconnect is ongoing, it will result in deserializing executor's address into socket address failed. The executorAddresses should save the executors' socket address only.