smp only used BatchNorm as normalization layer.
We should add the option of choosing other normalization layers, namely:
GroupNorm
LayerNorm
InstanceNorm
and their variations.
One small problem arise: while BatchNorm and InstanceNorm require parameters which can be easily inferred from the specific used architecture, GroupNorm and LayerNorm requires specific parameters to be defined, namely num_groups for GroupNorm, and normalized_shape for LayerNorm.
Probably normalized_shape can somewhat be inferred from the input size with a series of precautions.
smp only used BatchNorm as normalization layer. We should add the option of choosing other normalization layers, namely:
and their variations.
One small problem arise: while BatchNorm and InstanceNorm require parameters which can be easily inferred from the specific used architecture, GroupNorm and LayerNorm requires specific parameters to be defined, namely
num_groupsfor GroupNorm, andnormalized_shapefor LayerNorm. Probablynormalized_shapecan somewhat be inferred from the input size with a series of precautions.