TypeError on scheduler.start (possible version conflict?)

[bschreck]

• A description of the bug

  I'm trying to submit a dask cluster on EMR, and the dask scheduler fails to start. Seems like dask-yarn is calling the API incorrectly. I'm wondering if somehow the environment is not the same on the scheduler node? I used the stock dask EMR bootstrap script to install dask prereqs on EMR: https://github.com/dask/dask-yarn/blob/master/deployment_resources/aws-emr/bootstrap-dask

• Steps to reproduce

  dask-yarn submit --worker-env ARROW_LIBHDFS_DIR=/usr/lib/hadoop/lib/native/ --client-env ARROW_LIBHDFS_DIR=/usr/lib/hadoop/lib/native/ --name testdaskyarnapp --deploy-mode remote start_dask.py --worker-memory 16GiB --scheduler-memory 16GiB --scheduler-vcores 4 --worker-vcores 4

• Relevant logs/tracebacks

Container: container_1564686111027_0004_01_000002 on [IP ADDRESS].internal_8041

LogType:dask.scheduler.log Log Upload Time:Thu Aug 01 20:12:53 +0000 2019 LogLength:768 Log Contents: Fontconfig error: Cannot load default config file Traceback (most recent call last): File "/mnt1/yarn/usercache/hadoop/appcache/application_1564686111027_0004/container_1564686111027_0004_01_000002/environment/bin/dask-yarn", line 11, in sys.exit(main()) File "/mnt1/yarn/usercache/hadoop/appcache/application_1564686111027_0004/container_1564686111027_0004_01_000002/environment/lib/python3.6/site-packages/dask_yarn/cli.py", line 412, in main func(**kwargs) File "/mnt1/yarn/usercache/hadoop/appcache/application_1564686111027_0004/container_1564686111027_0004_01_000002/environment/lib/python3.6/site-packages/dask_yarn/cli.py", line 316, in scheduler scheduler.start(addr) TypeError: start() takes 1 positional argument but 2 were given End of LogType:dask.scheduler.log

• Version information

  Please include version information for the following:

  • Python version: 3.6.9
  • Dask-Yarn version 0.6.2
  • Hadoop version, and distribution: Amazon 2.8.5, EMR 5.24.1

[bschreck]

Ah... distributed released a new version yesterday. I'll hardcode and downgrade for now

[jcrist]

Yeah, this is a duplicate of #87, closing. Hope to get a new release out by EOD to fix this, but hardcoding your versions should work for now.

[jcrist]

Version 0.7.0 has been released, and is compatible with the new distributed release. Available on PyPI and shortly on conda-forge.