dask / knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
http://knit.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
53 stars 10 forks source link

Fix for -1000 case #114

Closed martindurant closed 6 years ago

martindurant commented 6 years ago

Fixes #111

martindurant commented 6 years ago

@jcrist , I believe this is correct. I don't see how tests could pass both before and after - perhaps the defaultFS on some systems points to the local disc, but in the particular image used by the tests, all paths are promoted to HDFS? I tested on stand-alone cluster which showed the -1000 error.

jcrist commented 6 years ago

I can confirm this fixes my issue. Also not sure how tests passed before.

martindurant commented 6 years ago

Merge and release? Could also include the apparently simple fix in #112 (again, can't realistically be tested)

jcrist commented 6 years ago

Merge and release sounds good to me. I'll leave #112 up to you, not sure what's best there.

martindurant commented 6 years ago

I think setting the memory request for the AM >>300MB is totally reasonable and harmless, since the typical minimum allotment is 1GB anyway. The only alternative fix for that issue is documentation of config changes.