search

liuzhuang13 / DenseNet

Densely Connected Convolutional Networks, In CVPR 2017 (Best Paper Award).
BSD 3-Clause "New" or "Revised" License
4.75k stars 1.07k forks source link

question about standardization #55

Open yuffon opened 5 years ago

yuffon commented 5 years ago

I find that in the training and testing phase, the dataset is standardized as a whole batch, including computing the mean and variance from the entire cifar10 dataset. However, when the model is deployed, the image is feed individually, how should we preprocess the image? What mean and variance should we use when the input image is of the same category but not included in the cifar10 dataset?

liuzhuang13 commented 5 years ago

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

yuffon commented 5 years ago

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

Yes, this is common in ML. What if the input image is not from the 10 categories in cifar10 dataset? If I want to do transfer learning or other tasks, how should I preprocess the data? I think it is more convenient with per image standardization. So I replaced the whole-batch standardization with per image standardization, the acc doesn't converge, untill that I add an BN layer without scale and shift variables at the start point of the densenet. I have not checked the final accuracy results in different scenarios.

liuzhuang13 commented 5 years ago

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

yuffon commented 5 years ago

In TensorFlow's official tutorial Resnet repository, they use per image standardization. https://github.com/tensorflow/models/tree/master/official/resnet

yuffon commented 5 years ago

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

If I make a new network that trained on cifar10, but I want to test the model on image out of the cifar10 dataset and observe the network behavior on the out-of-distribution data. How should I normalize the data?

liuzhuang13 commented 5 years ago

I think the common practice is to just normalize the data as the way it is normalized during training (and using the training stats).