search

ManifoldRG / NEKO

In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks
https://discord.gg/brsPnzNd8h
GNU General Public License v3.0
43 stars 10 forks source link

Wandb tracking issue for distributed runs launched with accelerate. #33

Open eihli opened 10 months ago

eihli commented 10 months ago

I noticed I was getting 4 wandb runs when I trained with accelerate launch on a machine with 4 GPUs. All 4 of the wandb runs include system statistics, like GPU temp. But only one of them includes the training/evaluation panels, because of our check for is_main_process.

All 4 of the runs highlighted in the image below are from a single accelerate launch train.py ... image

It looks like the way to use wandb with Accelerator is to use the log_with argument when instantiating the accelerator: accelerator = Accelerator(log_with="wandb") and then call accelerator.init_trackers under a condition of is_main_process.

This issue has some discussion on the 🤗 forums: https://discuss.huggingface.co/t/multiple-wandb-outputs/21394