Is there a way to visualize different worker runs separately for a distributed Tensorflow run

tensorflow / tensorboard

TensorFlow's Visualization Toolkit

Apache License 2.0

6.68k stars 1.65k forks source link

Is there a way to visualize different worker runs separately for a distributed Tensorflow run #640

Open ghost opened 6 years ago

ghost commented 6 years ago

I am running Tensorboard on distributed training logs. I can see operations on different parameter servers. They are color coded differently with device placement toggle. But, I can’t see operations run on different workers separately. Looks like everything has got merged in to worker 0.

Is there any way, to take a look at what different workers are doing in distributed settings?

MtDersvan commented 6 years ago

Also, instead of a list of devices it shows a single black 'unknown device'. This happened after 0.1.4 release.

imtarit commented 6 years ago

Also, instead of a list of devices it shows a single black 'unknown device'. This happened after 0.1.4 release.

I am having similar issue. What could be the possible cause? capture

Guneetkaur03 commented 5 years ago

I too have the same issue. I have made a cluster with two parameter servers, 2 workers and 1 master. Tensorboard shows me only two PS and no worker device. Screenshot from 2019-03-22 00-46-04