acc decrease from 0.8xxx to 0.5xxx when normalized in FashionMNIST

[vxgu86]

I test the FullyConnectedRegularized with normalized set to True, but the acc decrease from 0.8xxx to 0.5xxx when normalized in FashionMNIST. You said that the FashionMNIST dataset is already normalized, but the normalized after normalized would have no effect from math perspective, am I right?

[vxgu86]

the code I used as follow:

train_valid_dataset=torchvision.datasets.FashionMNIST(root=dataset_dir,
                                                      train=True,
                                                      transform=None, #transoforms.ToTensor(),
                                                      download=True)

nb_train=int((1.0-valid_ratio)*len(train_valid_dataset))
nb_valid=int(valid_ratio      *len(train_valid_dataset))
train_dataset,valid_dataset=torch.utils.data.dataset.random_split(train_valid_dataset,[nb_train,nb_valid])

test_dataset=torchvision.datasets.FashionMNIST(root=dataset_dir,
                                               transform=None,#transforms.ToTensor(),
                                               train=False)

class DatasetTransformer(torch.utils.data.Dataset):
    def __init__(self,base_dataset,transform):
        self.base_dataset=base_dataset
        self.transform=transform

    def __getitem__(self, index):
        img,target=self.base_dataset[index]
        return self.transform(img),target

    def __len__(self):
        return len(self.base_dataset)

train_augment_transform=transforms.Compose([
    transforms.RandomHorizontalFlip(0.5),
    transforms.RandomAffine(degrees=10,translate=(0.1,0.1)),
    transforms.ToTensor()
])

num_threads=0  #use 4 cpu threads
batch_size=128 #minibatch of 128
normalize=False
if normalize:
    normalize_dataset=DatasetTransformer(train_dataset,transforms.ToTensor())
    normalize_loader=torch.utils.data.DataLoader(dataset=normalize_dataset,
                                                  batch_size=batch_size,
                                                  num_workers=num_threads)

    mean_train_tensor,std_train_tensor=compute_mean_std(normalize_loader)
    train_augment_transform=transforms.Compose([
        train_augment_transform,
        transforms.Lambda(lambda x:(x-mean_train_tensor)/std_train_tensor)
    ])
    print("----normalize----")

train_dataset=DatasetTransformer(train_dataset,train_augment_transform)
valid_dataset=DatasetTransformer(valid_dataset,transforms.ToTensor())
test_dataset =DatasetTransformer(test_dataset ,transforms.ToTensor())

[jeremyfix]

When you say acc drops down to 0.5 , which acc do you speak about ? The train, valid or test metrics ?

May I suggest you try without data augmentation for computing the mean/std ?

[jeremyfix]

Here is what I get running the solution I provide, with or without data augmentation, and without normalization:

➜  00-pytorch-FashionMNIST git:(master) ✗ python3 train.py --num_workers 7 --data_augment --model fcreg
[....]
Epoch 9
 [============================ 375/375 ===========================>]  Step:       29ms | Tot:    12s94ms | Loss : 0.5831, Acc : 0.7826                 
 Validation : Loss : 0.4696, Acc : 0.8234
 Test       : Loss : 0.5069, Acc : 0.8149

➜  00-pytorch-FashionMNIST git:(master) ✗ python3 train.py --num_workers 7 --normalize --model fcreg
[....]
Epoch 9
 [============================ 375/375 ===========================>]  Step:        8ms | Tot:    3s900ms | Loss : 0.3349, Acc : 0.8867          [0/157]
 Validation : Loss : 0.4348, Acc : 0.8538
 Test       : Loss : 0.4854, Acc : 0.8444

➜  00-pytorch-FashionMNIST git:(master) ✗ python3 train.py --num_workers 7 --data_augment --normalize --model fcreg 
[...]
Epoch 9
 [============================ 375/375 ===========================>]  Step:        6ms | Tot:    3s710ms | Loss : 0.6617, Acc : 0.7667                 
 Validation : Loss : 0.5579, Acc : 0.8143
 Test       : Loss : 0.5865, Acc : 0.8034

[vxgu86]

I am only testing the val metrics acc after each epoch and the test acc, from the code above, I did not compute the mean and std with augmented dataset，right? it's loaded from train_dataset without augmented.

what I get is that: for linearNet, the best test acc 0.8420 is from the no augment and no normalized version with augment, the test acc is 0.7322, with augment and normalization, the test acc is 0.4602. it's really big margin among the three accs, may I ask, what's the problem?

[jeremyfix]

You are right, I looked at your code too quickly, you compute the normalization from the original data.

It is not obvious to me, from the code you provide where there might be an issue.

The code you provide for building up the dataloaders is really similar to the one I provide as a a possible solution and as I tested above, I do not observe the gap you mention. In particular, with data augmentation, normalization and the FullyConnectedRegularized model, I do get 84% on the test set after 9 epochs.

Maybe you should share more than the snippet of code you provide, specify how many epochs you ran for.

[vxgu86]

I uploaded it here https://github.com/vxgu86/DL_classifier_helpme you can test it by running python stepbystep.py thanks, It has ghosted me for days.

[vxgu86]

@jeremyfix hello,any way to help me out?

[jeremyfix]

I get your bug. When your normalize, the normalization function should also be applied to the valid_dataset and test_dataset which is not what you are doing. I did not notice that from your first post where we see it actually.

Therefore, you must replace :

valid_dataset=DatasetTransformer(valid_dataset,transforms.ToTensor())
test_dataset =DatasetTransformer(test_dataset ,transforms.ToTensor())

by something like

test_transform  = transforms.ToTensor()
if normalize:
    test_transform=transforms.Compose([
        test_transform,
        transforms.Lambda(lambda x:(x-mean_train_tensor)/std_train_tensor)
    ])

valid_dataset=DatasetTransformer(valid_dataset, test_transform)
test_dataset =DatasetTransformer(test_dataset , test_transform)

And then you get the metrics you expect.

[jeremyfix]

@vxgu86 Did that solve your issue ? If so, would you mind closing the issue ?

[vxgu86]

sorry I did get some trouble from the COVID-19. but I remember that you mentioned somewhere that the dataset has already been normalized. then the normalization on train dataset should have no effect, neither on the test and valid dataset. is this thought wrong?

[vxgu86]

@jeremyfix hello are you there?

[jeremyfix]

@vxgu86 More or less there, from time to time;

If I'm not incorrect, a quite glance at a comment above and I'm not sure that normalization does have a significant effect indeed; That we do not get the sames results from runs to runs is true, but that's because of the stochasticity in the initialization;

Now, an effect that I observed in the past was like the one in the tutorial on keras . You can compare the metrics between the image "Logistic regression without normalization of the input" and "Logistic regression with input standardization" and you will see that we have a strong effect of standardization on the mean and standard deviation of the metrics even on the training set. On the specific cases of the figure, this is actually a problem because the problem at hand is convex and different losses on the training set is not at all expected. All that mean is that the optimization problem is not solved the appropriate way and actually explained by the fact that, with input not standardized, the optimization problem, although convex, might be bad conditioned (the slope of the loss might be really steep in some directions compared to others);

[jeremyfix]

@vxgu86 Should we consider your question answered ?

[vxgu86]

sorry I got it, thanks