Closed mdip226 closed 2 years ago
There could (should, but now, or later?) be a command line flag that specifies whether the user is using their own dask setup in a terminal or whether they want to use the dask python instantiation scheme. Which should take precedence is what I'm wondering
There could (should, but now, or later?) be a command line flag that specifies whether the user is using their own dask setup in a terminal or whether they want to use the dask python instantiation scheme. Which should take precedence is what I'm wondering
It's a kind of tricky question, but the way I'm thinking at the moment is that app.py
can be the bare-bones method which will just connect to whatever address you give it for the cluster, defaulting to localhost:8786
, and startup.py
can be the 'just work` method for people who just want to compute without having to set up a cluster. It's tricky because there really are two different ways to do this, and the dask python API's main use case seems to be for writing these kinds of startup scripts, at least in part.
I realize now some of the changes I proposed initially turned out to be unnecessary! Can leave alot of things like they were initially. I had made some mistaken assumptions in what was going on and where in the startup script.
LocalCluster
attribute toDaskDelegateConfig
. I think this is good because, along with the other changes indaskdelegate.py
, accounts for the two ways thatdistributed.Client()
can be instantiated. One with a cluster object, one with just an address. Seems like a dask design choice that we have to account for somehow and this was the best way I could figure to do that while still keeping things simple but I'm open for discussionDaskDelegateConfig
are chosen based upon the Constructor's default arguments now. There seems to be a small error in the dask code base where they say that the default port is 8786, but the constructor defaults to 0, which is a random port. Going with the choice I made had other consequences (since the CL dask default port really is 8786, not 0) so I moved the setting of the default values intoDaskDelegate
's constructor in order to make it workable for people using either the python or command line versions of dask. I think this solved the problem but it's possible this issue may pop up later? In general I think it just has to do with the differences between the python and CL versions ofdask.distributed