FSD50K - validating on eval data

[Ludvig-Joborn]

Hi! First off, excellent work with the module. It's showing great results so far in my project. I'm having trouble, however, with an experiment. I am trying to fine-tune and train the model on subsets (3k samples for training and validating) and have created hdf5 files for that. The paths in config.basedatasets are corrected for this.

The problem that I run into is that when I run the command: python ex_fsd50k.py evaluate_only with passt_s_swa_p16_s16_128_ap473 the program uses the evaluation data for validation. I confirmed this by making a change in fsd50k/dataset.py:

def __len__(self):
    if self.hdf5_file == "audioset_hdf5s/mp3/FSD50K.eval_mp3.hdf":
        return 300
    return self.length

which affects the number of validation batches.

I really don't understand what is going on. Isn't the model supposed to validate on the validation data?

Kindest regards, Ludvig.

[kkoutini]

Hi! thank you! The evaluate_only command get the validation dataset here, you can edit this line to load any dataloader

val_loader = get_my_loader()

If you want to validate on another dataset, you can create for exmaple a custom loader (like this one )

get_my_loader = ex.datasets.eval.iter(DataLoader, static_args=dict(worker_init_fn=worker_init_fn),
                                        validate=False, batch_size=10, num_workers=16,
                                        dataset=CMD("/basedataset.get_my_eval_set"))

you need however to define the function that gets the dataset /basedataset.get_my_eval_set which can be similar to this

@dataset.command
def get_my_baseeval_set(my_eval_hdf5="path_to_my_hdf.hdf", variable_eval=None):
    if variable_eval:
        print("Variable length eval!!")
        ds = AudioSetDataset(my_eval_hdf5, clip_length=None)
    else:
        ds = AudioSetDataset(my_eval_hdf5)
    return ds

@dataset.command
def get_my_eval_set(normalize):
    ds = get_my_baseeval_set()
    if normalize:
        print("normalized test!")
        fill_norms()
        ds = PreprocessDataset(ds, norm_func)
    return ds

[Ludvig-Joborn]

First off; thank you a lot for answering! Second; forgive me, I made a silly mistake and wrote the wrong command in my earlier post. The command I used was

python ex_fsd50k.py with  passt_s_swa_p16_s16_128_ap473

and it makes use of the evaluation dataset for validation, which confuses me. I found this by changing the dataset length as described earlier. Is there something I'm misunderstanding?

[kkoutini]

Hi! yes the default behaviour is that the model is evaluated on both datasets. You can change that here by setting validate=False or adding datasets.eval.validate=False to your command:

python ex_fsd50k.py with  passt_s_swa_p16_s16_128_ap473 datasets.eval.validate=False

[Ludvig-Joborn]

Hi again. When I try to run the program as described above, I get this error:

ERROR - fsd50k - Failed after 0:03:31!
Traceback (most recent calls WITHOUT Sacred internals):
  File "ex_fsd50k.py", line 391, in evaluate_only
    res = trainer.validate(modul, val_dataloaders=val_loader)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 883, in validate
    results = self.fit(model)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 474, in fit
    self.dispatch()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 513, in dispatch
    self.accelerator.start_evaluating(self)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/accelerators/accelerator.py", line 95, in start_evaluating
    self.training_type_plugin.start_evaluating(trainer)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 139, in start_evaluating
    self._results = trainer.run_evaluate()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 745, in run_evaluate
    eval_loop_results, _ = self.run_evaluation()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 700, in run_evaluation
    deprecated_eval_results = self.evaluation_loop.evaluation_epoch_end()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py", line 187, in evaluation_epoch_end
    deprecated_results = self.__run_eval_epoch_end(self.num_dataloaders)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py", line 222, in __run_eval_epoch_end
    eval_results = model.validation_epoch_end(eval_results)
  File "ex_fsd50k.py", line 216, in validation_epoch_end
    avg_loss = torch.stack([x[net_name + 'val_loss'] for x in one_outputs]).mean()
  File "ex_fsd50k.py", line 216, in <listcomp>
    avg_loss = torch.stack([x[net_name + 'val_loss'] for x in one_outputs]).mean()
TypeError: string indices must be integers

I've spent quite some time trying to make it work but can't get it right. There seems to be some trouble in validation_epoch_end. You don't happen to know what needs to be changed?

[Ludvig-Joborn]

I think I've got it working now by copying the code for validation_epoch_end from ex_audioset.py, so everything should be settled now!