search

YuanGongND / ssast

Code for the AAAI 2022 paper "SSAST: Self-Supervised Audio Spectrogram Transformer".
BSD 3-Clause "New" or "Revised" License
365 stars 61 forks source link

For discriminative loss, is the true NCE batch size the number of masked patches? #26

Open hillup opened 1 year ago

hillup commented 1 year ago

image In this piece of code, it seems that the loss is calculated at the granularity of samples.

hillup commented 1 year ago

So even if you increase the number of gpus, contrastive learning will not see more negative examples.

YuanGongND commented 11 months ago

For discriminative loss, is the true NCE batch size the number of masked patches?

In line 347 in your screenshot, NCE is accumulated to all batch of samples, but the negative samples are all from the same spectrogram. I.e., say B=12 (you have 12 spectrograms in a batch), each spectrogram has 512 patches and you mask 400 of them. Then the negative samples is always 400-1=399, but NCE won't update until it goes through all 12 spectrograms.

So even if you increase the number of gpus, contrastive learning will not see more negative examples.

The negative samples will always be #masked_patches-1.