It seems that in many window-based segmentation/regression use cases, either settingis_training=True or setting use_local_stats=True in the batch normalisation layer at inference time leads to better performance than those with NiftyNet's default batch norm options.
Therefore this ticket proposes to add a use_local_stats option in the config file, and make use_local_stats=True as the default for segmentation application with batch norm.
Specifically, in the current implementation (v0.4.0) the default values are:
training: is_trainign=Trueuse_local_stats=True
inference: is_training=Falseuse_local_stats=True
which uses running averages from training at inference time, following Ioffe and Szegedy (2015)https://arxiv.org/abs/1502.03167.
The proposed change will use these values by default:
training: is_trainign=Trueuse_local_stats=True
inference: is_training=Falseuse_local_stats=True
which will effectively follow Ulyanov et al. (2017)https://arxiv.org/abs/1607.08022.
It seems that in many window-based segmentation/regression use cases, either setting
is_training=Trueor settinguse_local_stats=Truein the batch normalisation layer at inference time leads to better performance than those with NiftyNet's default batch norm options.Therefore this ticket proposes to add a
use_local_statsoption in the config file, and makeuse_local_stats=Trueas the default for segmentation application with batch norm.Specifically, in the current implementation (v0.4.0) the default values are:
is_trainign=Trueuse_local_stats=Trueis_training=Falseuse_local_stats=Truewhich uses running averages from training at inference time, following Ioffe and Szegedy (2015) https://arxiv.org/abs/1502.03167.The proposed change will use these values by default:
is_trainign=Trueuse_local_stats=Trueis_training=Falseuse_local_stats=Truewhich will effectively follow Ulyanov et al. (2017) https://arxiv.org/abs/1607.08022.This ticket is related to https://github.com/NifTK/NiftyNet/issues/280, https://github.com/NifTK/NiftyNet/issues/285, https://github.com/NifTK/NiftyNet/pull/282.
thoughts?
Probably to be addressed with #282