search

lunarlab-gatech / MorphoSymm-Replication

Tools for exploiting Morphological Symmetries in robotics - Replicating Results from RSS 2023 paper
https://danfoa.github.io/MorphoSymm/
0 stars 0 forks source link

Validation Set is not consistent when varying the train ratio. #1

Open DanielChaseButterfield opened 3 weeks ago

DanielChaseButterfield commented 3 weeks ago

Upon reviewing the code in umich_contact_dataset.py, it appears that the validation set changes if the train_ratio changes, even if the val_ratio stays the same.

DanielChaseButterfield commented 3 weeks ago

For now, although this changes the validation set; the validation set is not used for Generating Figures 4-Left or Figures 4-Center, as they use the train and test sets.

Therefore, we can run that part of the experiment without fixing this bug.

Additionally, since the train ratio doesn't vary for Figure 4-right experiment, this also isn't an issue there either, as the validation set will be consistent in that scenario.

DanielChaseButterfield commented 3 weeks ago

Correction, I believe I was wrong in my previous comment. The file train_supervised.py uses validation loss (val_loss) in order to run the Early Stopping mechanism. This could change the final train losses and test f1-scores of Figures 4-left and center, since the validation set is inconsistent between train ratios.

Therefore, I probably do need to fix this bug in order to replicate that experiment.

However, our trained models of CNN, CNN-aug, and ECNN for 0.85 train ratio should be fine, as they all have the same validation set, and its most likely the last 15% of the dataset.