Linkerbrain / fact-ai-2021-project

2 stars 0 forks source link

Experiments To-do List #10

Open Linkerbrain opened 2 years ago

Linkerbrain commented 2 years ago

Minimal Experiments that should be run:

Reproduce results in paper

[x] Search Algorithm Cifar100, ResNet20 (~6 hours, lodewijk's pc) [Lodewijk]
[x] Train Cifar100, ResNet20, No Policy (~2 hours, lisa) [Alfonso]
[x] Attack Cifar100, ResNet20, No Policy (~10 hours, lisa) [Alfonso]
[x] Train Cifar100, ResNet20, Their Hybrid (3-1-7, 43-18-18) (~2 hours, lisa) [Alfonso]
[x] Attack Cifar100, ResNet20, Their Hybrid (3-1-7, 43-18-18) (~10 hours, lisa) [Alfonso]

Generalised to dataset not mentioned in paper) blocked by Implement new Dataset Story [Lucas]

[ ] Search Algorithm mnist, ResNet20 (~6 hours, lodewijk's pc)
[ ] Train mnist, ResNet20, No Policy (~2? hours, lisa)
[ ] Attack mnist, ResNet20, No Policy (~6? hours, lisa)
[ ] Train mnist, ResNet20, Their Hybrid (3-1-7, 43-18-18) (~2? hours, lisa)
[ ] Attack mnist, ResNet20, Their Hybrid (3-1-7, 43-18-18) (~6? hours, lisa)

Extra Experiments to run:

Cifar100 + different model

alvitawa commented 2 years ago

[x] Train Cifar100, ResNet20, Their Hybrid (3-1-7, 43-18-18) (~2 hours, lisa) [Alfonso]

Training log for Cifar100, with Hybrid policy and both augmented policies.

https://gist.github.com/alvitawa/1070214fa386a34213895f53b634a21d

It prints the augment indices like 43-18-18 in addition to a list of augmentations like: [RandomCrop(size=(32, 32), padding=4), RandomHorizontalFlip(p=0.5), <benchmark.comm.sub_transform object at 0x14649711dac0>]

Do not be mistaken, the first two augmentations in this list are the preprocessing of the data, the actual augmentations provided by the indices are combined into the last element of that list.