Misguiding wording in documentation when dealing with batch normalization

[brando90]

In the documentation it says:

In test mode, dropout and batch-normalization are bypassed

it says it bypasses the batch-normalization block which is not consistent with what the code does or what the paper says to do during test. I suggest to re-word that because it makes using the library harder to use.

See what it actually does here:

https://github.com/vlfeat/matconvnet/blob/master/matlab/%2Bdagnn/BatchNorm.m

  methods
    function outputs = forward(obj, inputs, params)
      if strcmp(obj.net.mode, 'test')
        outputs{1} = vl_nnbnorm(inputs{1}, params{1}, params{2}, ...
                                'moments', params{3}, ...
                                'epsilon', obj.epsilon) ;
      else
        outputs{1} = vl_nnbnorm(inputs{1}, params{1}, params{2}, ...
                                'epsilon', obj.epsilon) ;
      end
    end

[lenck]

Hi, sorry, but this is effectively a bypass - the batch normalisation is not being applied, as the moments are being fixed and are not approximated out of the current batch, no?

[brando90]

I found the word bypass confusing because I sort of interpreted it as completely removing the BN block during inference. Its true that other statistics are used during inference but I personally find it confusing its bypassing BN. Maybe a few more words could be nice to clarify it in the docs? idk just trying to help.