New Benchmarks

[guarin]

We are in the process of updating our benchmark code and adding benchmarks on the ImageNet dataset. The goal of this issue is to coordinate and track progress of the update.

The new benchmark code lives in lightly/benchmarks and initial focus is on reference implementations for the various SSL methods following the settings from the respective papers. We'll limit training to 100 epochs and small batch sizes (2x128) for now due to limited compute resources.

Note: The benchmarking code is very much work in progress, there will be frequent and breaking changes.

Implemented Methods

• [ ] #1452
• [ ] W-MSE benchmark
• [ ] SSL-EY benchmark PR
• [x] SimCLR #1162
• [x] #1230
• [ ] #1255
• [x] #1252
• [x] #1287
• [x] #1288
• [x] #1323
• [ ] #1368
• [ ] #1369
• [x] #1303
• [ ] #1370
• [ ] #1371
• [x] #1350
• [ ] #1372
• [ ] #1373
• [ ] #1374

Contribute

Contributions are very welcome! You can help us with the following:

• Add one of the existing models to the new benchmarks. For this you can follow the SimCLR example.
• Let us know if you spot any mistake in the models or benchmarking code.
• Run benchmarks for different epochs / batch size.

Let us know if you are working on a model and we can help you out with reviews/compute :)

[ibro45]

Will you be creating a tensorboard with training losses of all these networks? May be useful when a user wants to get an idea of how the loss behaves during training.

[guarin]

Hi, yes this is the goal :)