search

dask / dask-yarn

Deploy dask on YARN clusters
http://yarn.dask.org
BSD 3-Clause "New" or "Revised" License
69 stars 41 forks source link

Allow forwarding CLI options #69

Open jcrist opened 5 years ago

jcrist commented 5 years ago

Some options on the distributed CLI should be settable via dask-yarn (e.g. --dashboard-address). There are a few ways we could accomplish this:

From https://github.com/jcrist/skein/issues/176.

mrocklin commented 5 years ago

I think that it would be useful to start thinking about a uniform way to specify things across all of the projects.

On Mon, Jun 17, 2019 at 4:56 PM Jim Crist notifications@github.com wrote:

Some options on the distributed CLI should be settable via dask-yarn (e.g. --dashboard-address). There are a few ways we could accomplish this:

  • Update distributed so that all parameters are also part of the configuration. Users could then provide them via env (or via configuration #42 https://github.com/dask/dask-yarn/issues/42).
  • Add an option to specify these parameters as part of the command

From jcrist/skein#176 https://github.com/jcrist/skein/issues/176.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask-yarn/issues/69?email_source=notifications&email_token=AACKZTGS7G4W6MPPTRNJ74TP26QYVA5CNFSM4HYXPJLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GZ5AHFQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTBRTTTHKC7BX2WRGQDP26QYVANCNFSM4HYXPJLA .

jcrist commented 5 years ago

I agree. I think a precursor to this would be making the CLI scripts in distributed more uniform in terms of argument parsing/config, and how processes are started/stopped.

Edit: misread your comment. I still think the above is important for yarn specific (since we need our own cli scripts), but common arguments for the cluster managers for specifying the worker/scheduler command would be good.