search

uber / RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Other
323 stars 101 forks source link

Unsupported shuffle manager #26

Closed danielhaviv closed 3 years ago

danielhaviv commented 3 years ago

Hi, I'm trying to use the RSS client with Spark 2.4.7 but it fails to register:

Error in SQL statement: SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 10.0.227.167, executor 118): ExecutorLostFailure (executor 118 exited caused by one of the running tasks) Reason: Unable to create executor due to Unable to register with external shuffle server due to : java.lang.UnsupportedOperationException: Unsupported shuffle manager of executor: ExecutorShuffleInfo{localDirs=[/local_disk0/spark-7fab8fdc-44a9-4259-ad5d-fd2c0551c2ec/executor-eeb9da1f-0869-4ecb-8374-6ededd031062/blockmgr-3e832fe3-cfd5-446a-bec0-5659f34d9be5], subDirsPerLocalDir=64, shuffleManager=org.apache.spark.shuffle.RssShuffleManager}
    at org.apache.spark.network.shuffle.ExternalShuffleBlockResolver.registerExecutor(ExternalShuffleBlockResolver.java:144)
    at org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.handleMessage(ExternalShuffleBlockHandler.java:115)
    at org.apache.spark.network.shuffle.ExternalShuffleBlockHandler.receive(ExternalShuffleBlockHandler.java:83)
    at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:180)
mabansal commented 3 years ago

Looks like you have enabled the external shuffle service, please set "spark.shuffle.service.enabled" to false

danielhaviv commented 3 years ago

That was it.

Thank you.