search

joblib / joblib-spark

Joblib Apache Spark Backend
Apache License 2.0
242 stars 26 forks source link

Driver executor disk space fills up with temp files #30

Open warwickhadley opened 3 years ago

warwickhadley commented 3 years ago

We have been successful in using this library with smaller parameter spaces however as the parameter space increases we are running out of space on the driver executor. It appears that temp files are created in the container for each job and are never removed.e.g. the folder /mnt/yarn/usercache/hadoop/appcache/application_1624531736036_0001/spark-9d614b58-4959-4b9e-a0ef-da525c803263/pyspark-6e9409d1-d631-41f0-a2b4-c3d9060c23d9 just fills up even if we increase the size of the volume mounted, it will eventually fill up and take down the driver.

I am not sure if this is a problem with joblib, joblik-spark or the spark configuration of our cluster. If anyone has any suggestions I would be grateful.

fritshermans commented 2 years ago

is there already a fix for this problem? it takes down the driver for me as well

arezki1990 commented 1 year ago

i have the same issue here on databricks any fix