Tibbers-Dev

[TibbersHao]

Revamp of the notebook

Description

This PR covers the recent changes of the dlsia integration with added features of Tiled I/O

Feature Upgrades

• Upgraded Tiled Dataset Class to include recon data, mask data and seg data for saving
• Ported codes from notebook to src with rearrangements of modules
• Write back the single 3-d array of segmented results as an array container with metadata associated with raw data uri
• Fixed bugs for arg parsing and code is able to run with command line settings
• Included a Makefile for testing purpose

To Do's:

• Needs to add make commands for MSDNet and TUNet3+ testing
• Better way to save losses and models (DvC integration with Xiaoya)

To Test:

Follow the instruction on README.

Note: UID needs to be changed if an container pre-exists in the Tiled Server, as currently there is no good way to delete containers.

[dylanmcreynolds]

resolves #2

[dylanmcreynolds]

We talked about writing the metrics out to the files system and perhaps accessing them in a browser

[taxe10]

This is really cool! The test case detailed in the README file did not work at my end, I believe $(PARAMETERS) is not defined in the Makefile. Alternatively, the test with test_msdnet_maxdil went through correctly. Some suggestions moving forward:

• Splitting training and inference processes
• Saving metadata into tiled: the model (network name) may not fully capture the description of the ML model that was used to generate this results. For example, I can have a TUnet with depth 3 and another with depth 4. Hence, I’d suggest maybe adding the other network parameters to the metadata or using the UUID of the segmentation instance.
• To avoid redefining train_segmentation from dlsia, we should think about adding a custom loss/metrics exporter in dlsia (compatible with dvc, file system, etc). We should chat with Peter about this.

[TibbersHao]

2 upgrades with latest commits:

1. fixed the typo of $(PARAMETERS) -> $(TUNET_PARAMETERS) as a reflection of new commands in Makefile
2. Added a Dockerfile and corresponding GitHub Workflow actions.

[TibbersHao]

Most of the comments have been addressed in the latest commit.

To Do:

• Separate inference out of training and change tiled_dataloader accordingly

• New testing commands

Projected Finish Time:

• Tuesday 02/20/24

[TibbersHao]

The recent commit addresses most of the comments above.

Key Feature Upgrades:

• Renamed RECON -> DATA
• Separated training (src/train.py) and inference (src/segment.py) into 2 processes
• Small bug/typo fix

Note:

• At the end of training process, network parameters are saved into .pt file according to model_path
• When inference requests are initiated, models are loaded with DLSIA's in house submodules (network type and parameter files above needed)

[TibbersHao]

Wrapping up this PR.

To address our previous discussion, @Wiebke , Changes that have been made:

1. Fixed typo and changed back to logging.info().
2. Deleted the apt-get line in Dockerfile, tested with docker build and image was built successfully.

[TibbersHao]

Side note:

Github actions of workflows have been added in one of the previous commits to automatically create the Dockerfile. It will need some attention to see if it works properly after merging.