User feedback during retraining of a model

[SilverViking]

Retraining a model takes a lot of time. We should

1. allow the user to stop the retraining progress
2. provide visual feedback of the segmentation with the current state of the retrained model and allow the user to stop retraining if the result is deemed good enough already.

One simple approach for 2. would be to add a hyperlink to the frontend which opens the Tensorboard, which would should the training progress. Training information is saved by the neuralnets library in data/models/[project e.g. EML]/[name of model e.g. test_run2]/events.out.tfevents. and _checkpoint.pytorch. Eventually we would probably want to incorporate information similar to tensorboard directly in the biosegment frontend itself. (Use MLflow?)

[JorisRoels]

I guess we would later need to be able to filter the test images from the events file and update the UI every now and then.

[JorisRoels]

• [x] Install neuralnets biosegment branch instead of master branch
• [x] Joris makes Biosegment-specific neuralnets branch
• [x] Frank implements train_net method in Biosegment
• [x] Joris ensures that test_epoch returns the complete segmentation of the test set
• [ ] Implement UI for showing test images

[JorisRoels]

The neuralnets library now has a separate branch biosegment that is specifically designed for this application. The train_net on this branch trains for one epoch and tests the model by segmenting the complete volume and computing the IoU. Over time, the model with the highest IoU will be saved in the log directory as best_checkpoint.pytorch.

A consequence of this branching is that it will no longer be possible to install the correct version of neuralnets through pip (this is connected to the master branch). We will have to install neuralnets by cloning it do a directory NN_DIR and include NN_DIR/neuralnets in the PYTHONPATH. I added this to the to do.

[SilverViking]

We will have to install neuralnets by cloning it do a directory NN_DIR and include NN_DIR/neuralnets in the PYTHONPATH. I added this to the to do.

There is no need for manually cloning the neuralnets library, we can have conda do this for us. See https://github.com/vibbic/biosegment/commit/bf2902b691f0e4b3dc6783b81ab700919cfc25d2

[berombau]

• [ ] take random sample and log this region to events log during training loop
• [ ] access events log image via backend
• [ ] visualize image in dash frontend retraining ui
• [ ] stop button for graceful shutdown of retraining

[SilverViking]

• [ ] ? Use tensorboard
• [ ] ? Quantitative metrics? IoU, accuracy, hausdorff distance, ...