Question about Eq. 7 - Githubissues

rickgroen / cov-weighting

Implementation for our WACV 2021 paper "Multi-Loss Weighting with Coefficient of Variations"

MIT License

50 stars 10 forks source link

Question about Eq. 7 #1

Closed RuyiLian closed 3 years ago

RuyiLian commented 3 years ago

Hi,

Thanks for your great work. I just have a question about Eq.7 in your paper

In your implementation (losses/covweighting_loss.py) you use:

self.running_S_l += (x_l - self.running_mean_l) * (x_l - new_mean_l)
running_variance_l = self.running_S_l / (self.current_iter + 1)
self.running_std_l = torch.sqrt(running_variance_l + 1e-8)

I don't understand how this corresponds to the original Eq. 7. Could you please explain it? Thanks!

rickgroen commented 3 years ago

Hi,

No problem! Originally the Welford's algorithm estimates the variance as: But we can rewrite:

However, this adds an increasingly small value due to the factor t, which is problematic when t gets large. The code makes use of sum of squares of differences from the mean to alleviate this problem. The paper equation and the one in the code end up doing the same, but the code implementation is a more numerically stable method of computing the standard deviation.

Hope it helps!

RuyiLian commented 3 years ago

Thanks for your reply! It really helps a lot!