[Meta Issue] (Wang, 2022) Table 1 Reproduction

[dgcnz]

Description

This issue concerns ONLY the reproduction of Wang 2022 table 1 without manually imposing equivariance (varying alpha, for that see meta issue #61)

Configurations

The models and the SmokePlume datasets have been ported. We need to add experiment files because each symmetry has different decay_rate, out_length, etc (see: https://github.com/Rose-STL-Lab/Approximately-Equivariant-Nets/blob/master/run.sh)

SmokePlume configs:

• [x] Table 1 Rotation: configs/data/wang2022/rotation.yaml (✅)
• [x] Table 1 Scale: configs/data/wang2022/scale.yaml (✅)
• [x] Table 1 Translation: configs/data/wang2022/translation.yaml (✅)

Model default configs (DO NOT MODIFY THESE FILES, other experiment files rely on the defaults set here):

• [x] ConvNet: configs/model/wang2022/convnet.yaml (✅)
• [x] RGroup: configs/model/wang2022/rgroup.yaml (❓)
• [x] RSteer: configs/model/wang2022/rsteer.yaml (✅)

Experiment configs:

• [x] Rotation:
  • [x] ConvNet: configs/experiment/wang2022/rotation/convnet.yaml (❓)
  • [x] RGroup: configs/experiment/wang2022/rotation/rgroup.yaml (❓)
  • [x] RSteer: configs/experiment/wang2022/rotation/rsteer.yaml (✅)
• [ ] Translation: (see Q3)
  • [ ] ConvNet: configs/experiment/wang2022/translation/convnet.yaml (❓)
  • [ ] RGroup: configs/experiment/wang2022/translation/rgroup.yaml (❓)
  • [ ] RSteer: configs/experiment/wang2022/translation/rsteer.yaml (✅)
• [ ] Scale: (see Q3)
  • [ ] ConvNet: configs/experiment/wang2022/scale/convnet.yaml (❓)
  • [ ] RGroup: configs/experiment/wang2022/scale/rgroup.yaml (❓)
  • [ ] RSteer: configs/experiment/wang2022/scale/rsteer.yaml (✅)

These configs have to be at least tested with trainer.fast_dev_run to ensure that the model even processes data correctly. This doesn't account for model checkpointing and early stopping, so we'll have to add tests to that. Examples can be found in the Makefile's command test_wang2022_table_1 which you can run make test_wang2022_table_1.

Example testing command:

python -m src.train experiment=wang2022/rotation/rgroup +trainer.fast_dev_run=True data.batch_size=8

Tasks

• [ ] Research if there's an easy way to:
  • [ ] Test that early stopping is working fine
  • [ ] Test that model checkpointing is working fine (checkpoint every 10 epochs)
• [ ] Make 1 run for x>10 (12 is fine) epochs and check that all metrics are being plotted, checkpoints are being saved and early stopping is not warning anything
• [ ] Add code to do grid search (we can use this

Questions

• Q1: Should we do grid search? (for the sunday draft probably not)
• Q2: Should we run our results on multiple seed? (for the sunday draft probably not)
• Q3: Should we reproduce translation and scale? (does this even work for convnet?)
• Q4: Do all methods use the same alpha? (if so notify)

Legend:

• (❓): Educated guesses were made to get the parameters
• (✅): Author's code had full reproducibility

[Nesta-gitU]

Actually alejandro said he didnt nececarily want to see the convnet results for the reproduction

[Nesta-gitU]

Issue: /usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/setup.py:187: GPU available but not used. You can set it by doing Trainer(accelerator='gpu').

[dgcnz]

Issue: /usr/local/lib/python3.10/dist-packages/lightning/pytorch/trainer/setup.py:187: GPU available but not used. You can set it by doing Trainer(accelerator='gpu').

is this on your laptop? @Nesta-gitU

[Nesta-gitU]

colab

[dgcnz]

done