Question about pretrain

[marmoi]

Hi, thank you for publishing wavcaps! it is really useful. I am trying to reproduce the results using HTSAT-BART using the code pretrain.py. But I got the following errors:

• in pretrain.yaml in json_files: the list contains all the datasets with _pretrain.json extension (but are not provided in this repo), I guess the _final.json ones provided here are the same.
• in pretrain_dataset.py when loading the json files, gives the error that AudioCaps/train.json and Clotho/train.json do not have the "duration" parameter.

I am doing something wrong? Thank you.

[XinhaoMei]

Hi, thanks for your interests!

First, the json_files should include the path to the json files of the data you want to use. Yes, they should be the ones we provided in the huggingface repo. Second, you could add the "duration" key for these two datasets whose value shoud be the length of the audio clip (in second). Because we will group audio clips with similar duration in a batch, this key will be used. However, this is only applicable to single-card training. If you want to use multi-card training, please set bucket=False when use the pretrain_dataloader.

I hope this is helpful!

[marmoi]

Hi, thank you for your reply! it was very helpful. I have another question regarding the results from table VI from the paper. For the dataset Clotho using HTSAT-BART (baseline) the model has been trained only with Clotho? or AudioCaps was used also for pretraining it and then finetuned with Clotho?

Thank you!

[XinhaoMei]

Hi, thank you for your reply! it was very helpful. I have another question regarding the results from table VI from the paper. For the dataset Clotho using HTSAT-BART (baseline) the model has been trained only with Clotho? or AudioCaps was used also for pretraining it and then finetuned with Clotho?

Thank you!

Hi, for baseline models, we trained them only on Clotho or AudioCaps, pretraining was not used.