Closed zainsarwar865 closed 6 days ago
In the global_vars.py file, the _set_tensorboard_writer function only gets set if the rank == world size - 1. However, this can be an issue if ranks are local and world size > number of gpus on one node.
global_vars.py
_set_tensorboard_writer
rank == world size - 1
Thus, I propose setting it to the master node.
In the
global_vars.pyfile, the_set_tensorboard_writerfunction only gets set if therank == world size - 1. However, this can be an issue if ranks are local and world size > number of gpus on one node.Thus, I propose setting it to the master node.