RuntimeError: No Devices Found when calling client.submit

[SophiaLcy]

I am currently running two workers on two different hosts. When I print the client information, I can see that both workers are successfully connected to the Dask scheduler. I believe I have sourced xrt correctly, but they are still unable to detect any devices when calling client.submit.

This can be successfully executed with the correct output.

  for i in range(len(pynq.Device.devices)):
          print("{}) {}".format(i, pynq.Device.devices[i].name))

but when running

  import platform, os

  if len(workers) != 2:
      print("Configure your Dask cluster with two workers")

  def verify_workers():
      import pynq
      import subprocess
      import os
      node_name = platform.node()
      device_name = pynq.Device.active_device[0]

      return node_name, device_name

  worker_0 = client.submit(verify_workers, workers=workers[0], pure=False)
  worker_1 = client.submit(verify_workers, workers=workers[1], pure=False)

  worker_check = [worker_0.result(), worker_1.result()]

  for w in worker_check:
      print('Worker name: {} | device name: {}'.format(w[0], w[1]))

It shows error:

I am afraid I do not know how to launch a worker correctly. Now I am running in this way

I would greatly appreciate your guidance on this matter.

[mariodruiz]

@SophiaLcy, the image is missing

[SophiaLcy]

sorry I was uploading the image which was really slow T_T

[mariodruiz]

Do you have PYNQ and XRT installed in both machines?

[SophiaLcy]

Do you have PYNQ and XRT installed in both machines?

yes when I type "which pynq" it show /rshome/*****/miniconda3/envs/xup-vitis/bin/pynq

And thank you for your quik reply!

[mariodruiz]

is this in both machines? Can you please comment about your hardware configuration?

[SophiaLcy]

I installed XRT on both machines. However, I installed PYNQ on an NFS directory, so they share the same one, and I only had to install it once. The host's hardware configuration is attached in the original comment; the other is as follows.

[mariodruiz]

What is the output when you print the client?

from dask.distributed import Client, get_client

client = Client(<your connection>)
client

[mariodruiz]

It's been a while since I have not touched that part, but this could be useful: https://docs.dask.org/en/latest/deploying-ssh.html

[SophiaLcy]

What is the output when you print the client?

from dask.distributed import Client, get_client

client = Client(<your connection>)
client

this is the output

[mariodruiz]

All the dask config looks good. Does dask have access to the same env as PYNQ?

Have you tried using PYNQ and checking if the device is visible? The issues seems like a env problem.