search

Abe404 / root_painter

RootPainter: Deep Learning Segmentation of Biological Images with Corrective Annotation
https://nph.onlinelibrary.wiley.com/doi/full/10.1111/nph.18387
Other
59 stars 17 forks source link

users should be warned when they try to start training when a model is already training #136

Open Abe404 opened 7 months ago

Abe404 commented 7 months ago

right now if a model is training and the user clicks 'start training' for a different project, training does not start (for the new project) as training is already running, but the user is not told the reason why training does not start.

They should at least be made aware that training was not started because 'training already running for project example_project' or similar.

Felipegalind0 commented 6 months ago

I’ll add some code to print a warning on the terminal in the trainer

Abe404 commented 6 months ago

That's a good idea, but in some cases users are running RootPainter purely from the client and only a sysadmin has access to the trainer.

There's a mechanism to send messages to the client that could be used. I'm also wondering if we could add something a little more obvious i.e some type of alert box that pops up in the client telling them that training is already running (the current messages can sometimes be missed, as they get overwritten by the next message).

Just to clarify: The issue is for users that start training for a project where training is already running on the server for a different project. This is an issue that comes up when multiple users are sharing a server.

I'm not 100% sure what the best solution for this is. Basically if users start training (from the client) when the server is already running training for a different user the client should at least let them know what the problem is somehow (in the client, as they might not see the server output).

Abe404 commented 6 months ago

This could also happen in a single user case.

For example, if a user follows the following steps:

  1. User starts training for their project.
  2. User opens a new project
  3. User starts training in the new project (Training won't work because trainer is busy with the previous project but the user doesn't get any feedback in the client).
Abe404 commented 6 months ago

Another option is we rethink how to give the user info about the server status via the client i.e have a custom module (GUI widget) that allows the user to stop/start the server, view server messages, get notifications/warnings/errors from the server etc without looking at the command line. This is a bit more involved and would require more discussion/design.