Experiments and Model Hyper-parameters Sweep on Slurm

[mmiakashs]

❓ Questions and Help

I was considering to conduct hyperparameters sweeps on slurm. I was following this tutorial: https://mmf.sh/docs/tutorials/slurm Here, it was trying to sweep the experiments related to hyper-parameters (batch size/ learning rate). However, is it possible to conduct both experiments related to hyper-parameters (listed in exp_config.yaml) as well as model related hyper-parameters (listed in model_config.yaml) using the same scripts?

For example:

import lib as sweep

from lib import hyperparam

def get_grid(args):
    # For list of args, run `python tools/sweeps/sweep_mmbt_hm.py --help`.

    return [
        # Single values (instead of list) remain constant
        hyperparam("model", "mmbt"),
        hyperparam("dataset", "hateful_memes"),
        hyperparam("exp_config", "projects/mmbt/configs/hateful_memes/exp_config.yaml"),
        hyperparam("model_config", "projects/mmbt/configs/hateful_memes/model_config.yaml"),
        # Grid sweep for 512 and 256
        # save_dir_key value is appended to the folder path for easy recognition
        hyperparam(
            "model.dropout", [0.1, 0.6], save_dir_key=lambda val: f"bs{val}"
        ),
        hyperparam(
            "training.batch_size", [512, 256], save_dir_key=lambda val: f"bs{val}"
        ),
        hyperparam(
            "optimizer.params.lr", [5e-5, 1e-5], save_dir_key=lambda val: f"lr{val}"
        ),
    ]
    # In the above case, sweep will be run over four combinations of batch
    # size and lrs, and folder name would mmbt.bsX.lrY.ngpuZ where
    # number of gpus will be specified via command line argument to this script

def postprocess_hyperparams(args, config):
    # Here you can post process a final config that is generated
    # to adapt some things
    pass

# When directly called through command line, run sweep's main function with
# provided grid configuration
if __name__ == "__main__":
    sweep.main(get_grid, postprocess_hyperparams)

Or could you please share any resource for sweeping both experiments and model-related parameters on slurm? Thanks

[vedanuj]

You can change the values of a hyperparam to a list for any model config parameters to sweep over those list values.

For example :

        hyperparam(
            "model_config.visual_bert.num_labels", [2, 4], save_dir_key=lambda val: f"num_labels{val}"
        ),

Does that answer your question?

[mmiakashs]

Thanks, @vedanuj for the explanation. I have very novice level questions: in the hyperparameters sweep, why do we need to add a prefix of _modelconfig in front of the _modelconfig hyperparameters, whereas we do not need to include the _expconfig in front of the exp_config parameters (in the above cases _batchsize and lr)? is it default behavior?

[vedanuj]

You should check this note about MMF configuration system to understand this in detail.

https://mmf.sh/docs/notes/configuration