[question] pytorch training

[monofo]

Hi, Now I want to make pytorch learn cifar10, but I get an error. This is my code and error

dname = "cifar10"
mname = "vgg16"
print('==> Preparing data..')
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)

bs= 16
in_channels=3
img_size=32
num_classes=10
x_test, y_test = get_dataset("cifar10")

data_gen = PyTorchDataGenerator(trainloader, size=len(trainset), batch_size=bs)
# Step 2: Create the modeld
print("===> prepare model")
# net = ResNet34()
if mname == "vgg16":
    model = VGG_CHEST('VGG16', in_channels, num_classes)
if mname == "resnet18":
    model = ResNet(BasicBlock, [2,2,2,2], in_channels, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4, nesterov=True)
nepoch = 30

classifier = PyTorchClassifier(model=model, loss=criterion,  
                            optimizer=optimizer, input_shape=(in_channels, img_size, img_size), nb_classes=num_classes)

print("lerning ...")
classifier.fit_generator(data_gen, nb_epochs=30)

predictions = classifier.predict(x_test)
accuracy = np.sum(np.argmax(predictions, axis=1) == np.argmax(y_test, axis=1)) / len(y_test)
print("Accuracy on benign test examples: {}%".format(accuracy * 100))

classifier.save(f"{mname}_{dname}", "./models/checkpoints/")

Traceback (most recent call last):
  File "train_cifar.py", line 186, in <module>
    classifier.fit_generator(data_gen, nb_epochs=30)
  File "./adversarial-robustness-toolbox/art/classifiers/pytorch.py", line 196, in fit_generator
    o_batch = torch.argmax(o_batch.to(self._device), dim=1)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

How can i fix?

[beat-buesser]

Hi @nquntan Thank you very much for using ART! I would be interested to find out why this error occurs. Could you please provide the python imports and PyTorch version that you have used for your script above?

[monofo]

$ python --version
3.7.3
$ python -c "import torch; print(torch.__version__)"
1.4.0

It seems that the target value was not set to an onehot, which was not good. However, in pytorch, the behavior was changed by changing the number of batch_size in predict.(test accuracy) That didn't happen with keras. Why is this?

[beat-buesser]

I would like to find out why too. I would like to try to run your script, could please also share your imports at the top of your script?

[monofo]

import argparse
import os
import random
import sys
from keras.utils import to_categorical
import numpy as np
import torch
import torch.backends.cudnn as cudnn
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
from art.classifiers import PyTorchClassifier
from art.data_generators import PyTorchDataGenerator
from art.utils import load_dataset, projection, random_sphere
from torch.optim.lr_scheduler import (CosineAnnealingLR, LambdaLR, MultiStepLR,
                                      ReduceLROnPlateau, StepLR)

"""
vgg.py
"""
import torch
import torch.nn as nn
import torch.nn.functional as F

cfg = {
    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 10)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)

class VGG_CHEST(nn.Module):
    def __init__(self, vgg_name, in_channels, num_classes):
        super(VGG_CHEST, self).__init__()
        self.features = self._make_layers(cfg[vgg_name], in_channels)
        self.avgpool = nn.AdaptiveAvgPool2d((7, 7))
        self.classifier = nn.Linear(512, num_classes)

    def forward(self, x):
        bs, _, _, _ = x.shape
        out = self.features(x)
        out = F.adaptive_avg_pool2d(out, 1).reshape(bs, -1)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg, in_channels):
        layers = []
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)

def test():
    net = VGG_CHEST('VGG11', 2, 1)
    x = torch.randn(2,1,229,229)
    y = net(x)
    print(y.size())

if __name__ == '__main__':
    test()

def get_dataset(dname="imagenet"):
    if dname == "imagenet":
        dataset = ImageNetValDataset("/home/kazuki/workspace/simple-blackbox-attack/image_net/", IMAGENET_TRANSFORM)
        dataloader = torch.utils.data.DataLoader(dataset, batch_size=10000,
                                                shuffle=False, num_workers=30)

    elif dname == "cifar10":
        dataset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=CIFAR_TRANSFORM)
        dataloader = torch.utils.data.DataLoader(dataset, batch_size=10000, shuffle=False, num_workers=2)

    images = next(iter(dataloader))[0].numpy()
    labels = next(iter(dataloader))[1].numpy()
    return images, labels

Here is my code.

[beat-buesser]

It looks like PyTorchDataGenerator is expecting one-hot-encoded labels. If you add these lines to your script it should work:

class OneHot:

    def __init__(self, num_classes):
        self.num_classes = num_classes

    def __call__(self, labels):
        y = torch.eye(self.num_classes)
        return y[labels]

target_transform = transforms.Compose([
    OneHot(num_classes=10),
])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train, target_transform=target_transform)

[monofo]

sorry for the late reply and Thanks for the solution.

[yuhaoban]

This really saves my day!