Shouldn't the val set use the train data augment? (but test always uses test)

[brando90]

Usually the val set is used to choose a model or even combine the train and val set. i.e. the val set comes from the train - test split. Also, augmentation helps for models to be better, so wouldn't it make more sense to have the val set version that might improve the model most?

With this logic wouldn't it make more sense to do:

        train_dataset.transform = train_data_transforms
        valid_dataset.transform = train_data_transforms
        test_dataset.transform = test_data_transforms

instead?

refs:

• https://stats.stackexchange.com/questions/320800/data-augmentation-on-training-set-only
• https://stats.stackexchange.com/questions/564146/does-it-make-sense-to-use-data-augmentation-on-the-validation-set-note-this-i?noredirect=1&lq=1

[seba-1511]

That's a good question, I think you could make the argument for using either train or test transforms for the validation tasks. The convention in the literature is to use test transforms, so we're sticking to that.

[brando90]

That's a good question, I think you could make the argument for using either train or test transforms for the validation tasks. The convention in the literature is to use test transforms, so we're sticking to that.

I think I agree that using the test augmentation makes more sense. The main reason I think this is that unless you are using the validation set as additional data to train using the train augmentation on the val set (or test set) doesn't help that much since that additional data is not really being used to change the weights. Instead (at least in my code) they are used for early stopping like stuff. Thus, train augmentation of val would increase the variance of the val loss which might make it harder to really choose a good model using the val set. I did post more things above that I've not read that carefully but I agree with seb that test transform on the validation set makes sense (unless the val and train and combined).

[brando90]

That's a good question, I think you could make the argument for using either train or test transforms for the validation tasks. The convention in the literature is to use test transforms, so we're sticking to that.

I think I agree that using the test augmentation makes more sense. The main reason I think this is that unless you are using the validation set as additional data to train using the train augmentation on the val set (or test set) doesn't help that much since that additional data is not really being used to change the weights. Instead (at least in my code) they are used for early stopping like stuff. Thus, train augmentation of val would increase the variance of the val loss which might make it harder to really choose a good model using the val set. I did post more things above that I've not read that carefully but I agree with seb that test transform on the validation set makes sense (unless the val and train and combined).

an example of the mi code is as follows:

    """Tasksets for mini-ImageNet benchmarks."""
    if data_augmentation is None:
        train_data_transforms = None
        test_data_transforms = None
    elif data_augmentation == 'normalize':
        train_data_transforms = Compose([
            lambda x: x / 255.0,
        ])
        test_data_transforms = train_data_transforms
    elif data_augmentation == 'lee2019':
        normalize = Normalize(
            mean=[120.39586422/255.0, 115.59361427/255.0, 104.54012653/255.0],
            std=[70.68188272/255.0, 68.27635443/255.0, 72.54505529/255.0],
        )
        train_data_transforms = Compose([
            ToPILImage(),
            RandomCrop(84, padding=8),
            ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4),
            RandomHorizontalFlip(),
            ToTensor(),
            normalize,
        ])
        test_data_transforms = Compose([
            normalize,
        ])
    else:
        raise('Invalid data_augmentation argument.')

    train_dataset = l2l.vision.datasets.MiniImagenet(
        root=root,
        mode='train',
        download=True,
    )
    valid_dataset = l2l.vision.datasets.MiniImagenet(
        root=root,
        mode='validation',
        download=True,
    )
    test_dataset = l2l.vision.datasets.MiniImagenet(
        root=root,
        mode='test',
        download=True,
    )
    if device is None:
        train_dataset.transform = train_data_transforms
        valid_dataset.transform = test_data_transforms
        test_dataset.transform = test_data_transforms
    else:
        train_dataset = l2l.data.OnDeviceDataset(
            dataset=train_dataset,
            transform=train_data_transforms,
            device=device,
        )
        valid_dataset = l2l.data.OnDeviceDataset(
            dataset=valid_dataset,
            transform=test_data_transforms,
            device=device,
        )
        test_dataset = l2l.data.OnDeviceDataset(
            dataset=test_dataset,
            transform=test_data_transforms,
            device=device,
        )

[brando90]

note: test set augmentation normally is only a normalization (and nothing else really).

[brando90]

That's a good question, I think you could make the argument for using either train or test transforms for the validation tasks. The convention in the literature is to use test transforms, so we're sticking to that.

I agree to use test transforms on validation. It reduces the variance on the validation and since your not fitting them anyway there is likely little benefit to early stop with complicated train data augmentation for valitation. Better to have a low variance estimate of an unknown distribution so to early stop more precisely.