search

uber / RemoteShuffleService

Remote shuffle service for Apache Spark to store shuffle data on remote servers.
Other
321 stars 100 forks source link

Not support Storing data on HDFS and dynamicAllocation? #61

Closed Lobo2008 closed 2 years ago

Lobo2008 commented 2 years ago

Hi， I found from the shuffle server log that shuffle data are stored on StreamServer:/tmp/StreamServer_xxxx, Does Uber RSS support to store shuffle data on HDFS now？ And when dynamicAllocation is true, It thrown Exceptions

spark.shuffle.service.enabled false spark.dynamicAllocation.enabled false

works fine

spark.shuffle.service.enabled true spark.dynamicAllocation.enabled false or true

ERROR YarnScheduler: Lost executor 1 on 10.174.18.9: Unable to create executor due to Unable to register with external shuffle server due to : java.lang.UnsupportedOperationException: Unsupported shuffle manager of executor: ExecutorShuffleInfo

spark.shuffle.service.enabled false spark.dynamicAllocation.enabled true

org.apache.spark.SparkException: Dynamic allocation of executors requires the external shuffle service. You may enable this through spark.shuffle.service.enabled.
mayurdb commented 2 years ago

Currently, RSS does not have a support to store the shuffle data in HDFS. This might be taken up in the future. Contributions are most welcome 🙂

Can you share your configs and complete stack trace? Doesn't look like the shuffle manager is set correctly