BatchNorm in test setting.

[Lan1991Xu]

Hi Thanks for your sharing code. I have noticed you have used different batch norm parameters in two domain. How about in the testing. I only find these code part related to batch norm setting in model_architecture.py, could you explain in detail. Thanks

[Britefury]

The reason you cannot find a specific implementation is that there is none :) It occurs as a side effect of how the batches are processed.

If data from both datasets is processed in a single batch:

combined_X = np.concatenate([source_dataset_X, target_dataset_X], axis=0)
combined_y = np.concatenate([source_dataset_y, target_dataset_y], axis=0)
combined_prediction = net(combined_X)
loss = criterion(combined_prediction, combined_y)

The batch norm stats (mean, std) will be computed using samples from both source and target

If processed separately:

source_prediction = net(source_dataset_X)
target_prediction = net(target_dataset_X)
source_loss = criterion(source_prediction, source_dataset_y)
target_loss = criterion(target_prediction, target_dataset_y)

Then batch norm will be applied to the source and target batches separately, using a different mean and std for each.

[Lan1991Xu]

Thanks for you explain. I think I just misunderstand in the test stage. Actually you used the mean and variance in the training, when apply BN in batch norm. So in the test stage, there are not separated “the mean and variance”. But then I come with an issue, why we need class DomainAdaptModule (nn.Module)? If we do not want to keep the statistic information for source and target separately. Can we directly use the default batch norm in pytorch. Is there any difference here? Thanks again.

From: Britefury notifications@github.com Reply-To: Britefury/self-ensemble-visual-domain-adapt-photo reply@reply.github.com Date: Monday, 15 October 2018 at 19:26 To: Britefury/self-ensemble-visual-domain-adapt-photo self-ensemble-visual-domain-adapt-photo@noreply.github.com Cc: Xu Lan x.lan@qmul.ac.uk, Author author@noreply.github.com Subject: Re: [Britefury/self-ensemble-visual-domain-adapt-photo] BatchNorm in test setting. (#3)

The reason you cannot find a specific implementation is that there is none :) It occurs as a side effect of how the batches are processed.

If data from both datasets is processed in a single batch:

combined_X = np.concatenate([source_dataset_X, target_dataset_X], axis=0)

combined_y = np.concatenate([source_dataset_y, target_dataset_y], axis=0)

combined_prediction = net(combined_X)

loss = criterion(combined_prediction, combined_y)

The batch norm stats (mean, std) will be computed using samples from both source and target

If processed separately:

source_prediction = net(source_dataset_X)

target_prediction = net(target_dataset_X)

source_loss = criterion(source_prediction, source_dataset_y)

target_loss = criterion(target_prediction, target_dataset_y)

Then batch norm will be applied to the source and target batches separately, using a different mean and std for each.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/Britefury/self-ensemble-visual-domain-adapt-photo/issues/3#issuecomment-429962801, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AVdxFneuHhR5ODqx-B_E568P8alAB1PRks5ulNMrgaJpZM4XT54J.