configure executor cores for spark-tensorflow-distributor to utilize multi-thread per worker

tensorflow / ecosystem

Integration of TensorFlow with other open-source frameworks

Apache License 2.0

1.37k stars 392 forks source link

configure executor cores for spark-tensorflow-distributor to utilize multi-thread per worker #168

Closed cmxcn closed 3 years ago

cmxcn commented 3 years ago

Hi I have one question regarding the correct configure to make spark-tensorflow-distributor to utilize multi-thread per worker. As pyspark will allocate tasks per core*executor, so there may be multiple workers running on one executor, any idea how to avoid this? Thanks in advance.

cmxcn commented 3 years ago

found config spark.task.cpus=core, issue closed