Dev Meeting: New dev avenues

[NMontanaBrown]

Subject of the issue

At the dev meeting we discussed that, post house-keeping, we should discuss new avenues for development of the codebase + registration functionalities, features beyond minor patches.

We suggested people read/investigate/have a think of papers to implement or new algorithms that could be of interest, or features that would significantly add to the repo.

Please add suggestions before the meeting here so we can have a look and productively discuss.

TODOs: Devs to bring 1 (or more!) suggestions to next dev meeting, tag here with links to paper/code + few lines on thoughts/ideas/directions....

[NMontanaBrown]

@YipengHu @RemiDelaunay @ZheMin-1992 @zacbaum @mathpluscode @acasamitjana @agrimwood @s-sd @ebonmati @MattClarkson @QianyeYang

[mathpluscode]

My reading list:

find representation for registration

• https://ieeexplore.ieee.org/document/8071026
• https://link.springer.com/chapter/10.1007/978-3-319-66179-7_40
• https://arxiv.org/abs/1911.04417

convert multimodel registration into monomodal

• https://arxiv.org/abs/1903.09331

GAN?

TBC

[acasamitjana]

Suggestions (no need to do them all, just brainstorming)

For completeness

• Losses: Mutual information (specially to compare with other intermodal registrations methods), Tukey Biweight (a more robust SSIM metric)
• Data augmentation: apart from the existing AffineTF, we can include DDF transformations as well
• Incorporate the option to resize deformation fields? The motivation behind is: (a) the associated notion of "control points" (e.g. if resize=4, we'll have control points every 4 pixels with linear interpolation to full size) (b) produces smooth DDF (c) DVF integration is faster.

Research directions

Cast intermodal registration (R) as an intramodal problem using image synthesis (I2I). A couple of interesting papers:

• [MICCAI20, https://arxiv.org/abs/2007.02790] R: They use a two streams to compute the final registration. I2I: CycleGAN with a structure preserving and identity losses. Training: not sure if I2I and registration networks are trained jointly. Evaluation: paired data.
• [IPMI 19, https://arxiv.org/abs/1903.09331 already mentioned by @mathpluscode]. R: they use a learning-based loss apart from the SSIM for the registration. I2I: MUNIT framework. Training: separate training of I2I and registration networks. Evaluation: paired data.

[s-sd]

New feature suggestion:

Automated hyperparameter tuning:

At MICCAI I saw a lot of interest in automated hyperparameter search/ tuning. The feature is available in several libraries but integrating it with parameters specified in the config file and with the deepreg train function is not straightforward. I suggest that we add some simple functionality that allows for an automated search procedure.

To start out we could add a simple function called something like tune_and_train which takes in a slightly modifed version of the config file. The modified config file can allow the user to specify either a range for a certain parameter or multiple values for a paramter. Then using random serach the tune_and_train function first selectes the best hyperparameters from its search and then uses these to train the network for a user defined number of epochs. To start out, we can offer this random search procedure which runs the train until a user defined number of epochs for each random combination of parameters that are selected from the user defined space for the parameters. Then the validation accuracy is used from each run to select the best combination which is then used to train the final model for a user defined number of epochs. As a starting point we can offer this functionality for a few parameters in the config file like batch size and number of channels. Then we can extend with other parameters and also other search algorithms, as required.

[YipengHu]

Mine is simple: add a pytorch version and as mentioned by Yunguan, the loader may be shared.

[NMontanaBrown]

Tagging @zacbaum, implement the Free Point Transformer? 😃

[RemiDelaunay]

+1 for pytorch What about adding a demo for registration of 2D images ? I'm not sure if Deepreg accepts 2D images as inputs already ?

[zacbaum]

Tagging @zacbaum, implement the Free Point Transformer? 😃

This was already going to be my suggestion... 😉

In addition to this, we would need to include a way of converting image label masks to pointsets (either on the fly, or as part of a pre-training pipeline).

[NMontanaBrown]

What are some standard registration packages out there we can point to? Would it be useful to self contain the package with standard implems of ICP/CPD/other classical algorithms? or include those packages and maybe add a module that can auto run this for validation as part of our "benchmarking" claim?

+1 for multi-uni modal stream @acasamitjana. It's close to home, very keen.

[NMontanaBrown]

Adding video from MEC, gif, to readme.md for nice graphs purposes.

[NMontanaBrown]

@agrimwood Adding information from tracker as inputs to the network to constrain reg, or as different inputs. So a module to do this/process txt files..?

[NMontanaBrown]

@ebonmati Consistent outputs for visualisation from the demos - we have all the image files but do not use them for anything/videos like the MEC one. @s-sd wrote a script to make gifs/videos, maybe it\s a case of adding this script/function into the demos to ensure consistent outputs, update readme with the video to make it more visually appealing.

[ebonmati]

Add a visual output for each demo showing results.

[s-sd]

@s-sd wrote a script to make gifs/videos

@ebonmati, more information (including the four types of visualisations that are supported) on the visualisation tool can be found here: https://github.com/DeepRegNet/DeepReg/blob/develop/docs/source/docs/visualisation_tool.md

It is currently in the develop branch