search

Nerogar / OneTrainer

OneTrainer is a one-stop solution for all your stable diffusion training needs.
GNU Affero General Public License v3.0
1.81k stars 153 forks source link

[Bug] Tensorboard is only available during the training run and not after it has finished #141

Open arkinson9 opened 10 months ago

arkinson9 commented 10 months ago

Tensorboard, started from the GUI, seems only to work while a training is running:

  1. If I press the "Tensorboard" button before I press the "Start Training" button, the Tensorboard Server seems not to start.
  2. Pressing "Tensorboard" after "Start Training" works, but:
  3. Immediately after a training is finished the connection to the tensorboard get lost. Even if you press the "Tensorboard" button again.

It would be great if you could use the Tensorboard easily via the GUI as long as the GUI window is opened.

O-J1 commented 10 months ago

Can confirm I can reproduce this with 100% rate. I thought this was intended behaviour lol

arkinson9 commented 10 months ago

Thank you for confirming.

SirTrippsalot commented 10 months ago

Can confirm I can reproduce this with 100% rate. I thought this was intended behaviour lol

It is actually completely intended behaviour. I have however created an initial PR to address this, instead tying the server stop to the launch of a new training run. There are some issues with it however, it was tied completely to the UI. I've got to re-work my approach, however at the moment I've got a few projects I'm working on.

Zueuk commented 8 months ago

Yeah this it is quite annoying. If only we could view these graphs inside the actual program window, instead of having to use a whole web browser just to view some graphs...

ppbrown commented 6 months ago

@Zueuk thats not what this bug is about at all. What you are referring to is an entirely different thing.