Resources generalisation in knit - Githubissues

dask / knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead

http://knit.readthedocs.io/en/latest/

BSD 3-Clause "New" or "Revised" License

53 stars 10 forks source link

Resources generalisation in knit #101

Closed martindurant closed 6 years ago

martindurant commented 6 years ago

files is a list for uploading, and should include any zipped python env. Things ending in .zip will be treated as archives with a name like the filename minus the .zip. Existing resources in hdfs can also be passed (no upload).
environment (set of key/values) is passed as a dictionary.
for now, dask launch command is simply derived from the environment passed, as before.

Meantime: changed application and client kill to atexit.

martindurant commented 6 years ago

Notes:

needs careful documentation
atexit method works in absolutely getting rid of hanging processes and applications, but often causes error messaging in the console at shutdown. This is presumably because everything it torn down at the same time; I don't know how to avoid it.

martindurant commented 6 years ago

Fixes #100

martindurant commented 6 years ago

@jcrist - is this how it's done? I'm not experienced with weakref.

martindurant commented 6 years ago

I stuck with the class attribute and method as, in my mind, this is better for possible subclassing - not that I expect much of that.