Open keewis opened 2 years ago
This is expected behaviour. The LocalCluster
object only exists as long as the Python process is still alive. So when the daskctl cluster create
command exits the cluster is gone.
I'm curious why you would want to create a LocalCluster
via dask-ctl
given how light weight they are?
mostly for experimentation with dask-ctl
, but I have also been using LocalCluster
in a pipeline with multiple commands that don't know how to create clusters.
In this case I'm also the author of those programs so I'm free to add support for that, but it would make both the CLI interface and the CLI code much more complex, so I'm a bit reluctant (and my own little tool solves this by blocking until the cluster is shut down). Maybe I should just use a different cluster type for this, though?
I'm guessing the core dump is not expected behavior?
Fair enough! I can't immediately think of a solution that would feel nice. Perhaps this highlights an opening for a detached process-based cluster manager, DaemonCluster
or something.
You're right that the core dump is not expected though.
I am noticing this core dump intermittently in CI too. I've opened #37 to track it separately.
I'm not sure if I'm doing something wrong (and I certainly don't understand the relation between
ProxyCluster
andLocalCluster
), but runningdoes not yield anything, and I also can't find any trace of the cluster process on my system. For reference, this works (even though there are some
KeyError: 'register-client'
errors:So I suspect
shutdown_on_close = False
does not work the way I thought it would.Even worse, creating the cluster using the CLI multiple times in a row (usually between 3-5 times over a relatively short time) results in
daskctl
crashing with apython
core dump (first time I've seen that). Not sure what went wrong, or whether it would be better to report that todistributed
instead.log of the crash
```pytb distributed.diskutils - INFO - Found stale lock file and directory '$PWD/dask-worker-space/worker-hdmzw82z', purging distributed.diskutils - INFO - Found stale lock file and directory '$PWD/dask-worker-space/worker-0snl7k9p', purging distributed.diskutils - INFO - Found stale lock file and directory '$PWD/dask-worker-space/worker-nb9_z2rd', purging distributed.diskutils - INFO - Found stale lock file and directory '$PWD/dask-worker-space/worker-kbdgg6l1', purging Created cluster 5fc5d1cd. Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join: Traceback (most recent call last): File ".../lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File ".../lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File ".../lib/python3.9/site-packages/distributed/process.py", line 218, in _watch_process assert exitcode is not None AssertionError Exception in thread AsyncProcess Dask Worker process (from Nanny) watch process join: Traceback (most recent call last): Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='Edit: versions: