Evaluate my own model

[WangHelin1997]

Hi authors! How can I evaluate my own trained model?

[kkoutini]

Hi! You can evaluate the model using this command: https://github.com/kkoutini/PaSST/blob/7a0d0a41b019cbae2547bf8e5383bc49e0f84713/ex_audioset.py#L352

which can be called like this:

python ex_audioset.py evaluate_only with trainer.precision=16 models.net.arch=passt_deit_bd_p16_384  trainer.use_tensorboard_logger=True -p

the loading model part is not implemented yet, but you can try for example:

@ex.command
def evaluate_only(_run, _config, _log, _rnd, _seed, preload_modul_path="path.pth"):
    # force overriding the config, not logged = not recommended
    trainer = ex.get_trainer()
    train_loader = ex.get_train_dataloaders()
    val_loader = ex.get_val_dataloaders()
    modul = M(ex)
    modul.val_dataloader = None
    trainer.val_dataloaders = None
    print(f"\n\nValidation len={len(val_loader)}\n")
    #here is the loading part
    ckpt = torch.load(preload_modul_path)
    modul.load_state_dict(ckpt)
    ###
    res = trainer.validate(modul, val_dataloaders=val_loader)
    print("\n\n Validtaion:")
    print(res)

now you can use in the config preload_modul_path to specify the path for your pretrained model:

python ex_audioset.py evaluate_only with trainer.precision=16 models.net.arch=passt_deit_bd_p16_384  trainer.use_tensorboard_logger=True preload_modul_path=/path/to/model.pth -p

[WangHelin1997]

It works!. Thanks a lot.