search

YuanGongND / ast

Code for the Interspeech 2021 paper "AST: Audio Spectrogram Transformer".
BSD 3-Clause "New" or "Revised" License
1.07k stars 205 forks source link

How to set the norm_stats for new dataset with pretrained model? #16

Closed joewale closed 2 years ago

joewale commented 2 years ago

Hi, YuanGong, run.py中的norm_stats对于每个数据集的参数是怎么设置的？依据什么？

image

YuanGongND commented 2 years ago

Hi there,

The norm stats are just the mean/std of the dataset, we calculate it using this script. Note that when we calculate the stats, the impact of mixup, SpecAug has already been considered, so you can see we input the audio_conf to the dataloader, which contains mixup, SpecAug information.

-Yuan

joewale commented 2 years ago

Thanks for your quick reply. I have understand how to do. Another question: audio_conf I used should be the same as it used in pretrained model ? I use the pretrained model as follows: https://www.dropbox.com/s/cv4knew8mvbrnvq/audioset_0.4593.pth?dl=1

YuanGongND commented 2 years ago

Hi there,

Only num_mel_bins of audio_conf needs to be the same as the pretrained model.

num_mel_bins: must be 128 to use the pretrained model. target_length: no need/ should not be the same with the pretrained model, it should be the temporal frame length of your dataset, e.g., if your audios are 1s, target_length should be 100. freqm and timem: no need/should not be same with the pretrained model, we suggest to mask a reasonable portion, but not all your data, e.g., if your target_length is 100, we suggest to set timem with 24. mixup: no need/should not be the same with the pretrained model, depend if you want to use mixup for your training. dataset: should not be the same as the pretrained model, but notice this is used here and there in traintest.py and run.py for loss function/evaluation strategy flags, please take a look and make modifications. mean and std: better to use the stats of your dataset, but it might be OK to just reuse the AudioSet stats. noise: depends on if you want to augment noise during training.

By the way, it is highly recommended to set audioset_pretrain=True and set input_tdim as your input length (in frames) when you initialize the AST model to use the pretrained model rather than manually use load_state_dict because the first will automatically adjust the positional embeddings for your input length, which is important.

-Yuan

joewale commented 2 years ago

Thanks, I have a try by your good advice.