search

AIforGoodSimulator / model-server

MIT License
11 stars 9 forks source link

Dask Fails as workers have different versions #171

Open kariso2000 opened 3 years ago

kariso2000 commented 3 years ago

Logs show :

/opt/python/3.7.9/lib/python3.7/site-packages/distributed/client.py:1129: VersionMismatchWarning: Mismatched versions found

+---------+---------------+---------------+---------------+
| Package | client        | scheduler     | workers       |
+---------+---------------+---------------+---------------+
| blosc   | 1.9.2         | 1.7.0         | 1.7.0         |
| dask    | 2021.01.1     | 2.30.0        | 2.30.0        |
| msgpack | 1.0.2         | 1.0.0         | 1.0.0         |
| numpy   | 1.18.5        | 1.18.1        | 1.18.1        |
| python  | 3.7.9.final.0 | 3.7.0.final.0 | 3.7.0.final.0 |
+---------+---------------+---------------+---------------+
kariso2000 commented 3 years ago

Need to update dask workers:

  1. Should only be done by admin staff and on local development machine
  2. Check your local .env file and ensure your DASK_SCHEDULER_HOST is set to the remote host and not 127.0.0.1
  3. Run upload.sh on your local host
  4. It is important that scheduler and workers are the same version. If you have made changes to your dev (client) machine that are required to run the code deployed on DEV then client must match scheduler and workers.
  5. You can check the logs on DEV via ssh web client :
cd /home/LogFiles
ls -lart
more 2021_02_01_RD501AC54710BD_default_docker.log

Note : the filename will be different, you need the last file returned by the ls -lart command.

  1. You can also check logs on the Dask web console - http://x.x.x.x/info/main/workers.html