search

Hydrospheredata / mist

Serverless proxy for Spark cluster
http://hydrosphere.io/mist/
Apache License 2.0
326 stars 68 forks source link

Cleanup worker directory #502

Open dos65 opened 6 years ago

dos65 commented 6 years ago

Should worker clean up its directory before stopping? Jars may take a lot of space on a disk, especially in combination with exclusive worker mode. Spark workers have special options for this - spark.worker.cleanup.* (disabled by default).