Conv-TasNet Cumulative Layer Norm Bug?

shouldn't lines 78-92 be

` step_sum = input.sum(dim=1) # -> (batch_size, T) cum_sum = torch.cumsum(step_sum, dim=1) # -> (batch_size, T)

cum_num = torch.arange(C, C(T+1), C, dtype=torch.float) # -> (T, ): [C, 2C, ..., T*C] cum_mean = cum_sum / cum_num # (batch_size, T) cum_var = (cum_sum - cum_mean)**2/cum_num

cum_mean = cum_mean.unsqueeze(dim=1) cum_var = cum_var.unsqueeze(dim=1)

output = (input - cum_mean) / (torch.sqrt(cum_var) + eps) * self.gamma + self.beta `

according to the Conv-TasNet paper?

My CumulativeLayerNorm1d is based on official implementation. You can compare it with mine here. If there is any possibility that I have misunderstood something, let me know in more detail.

compared to the official implementation, your code writes cum_var=cum_squared_mean - cum_mean*2 while the official code writes cum_var = (cum_pow_sum - 2cum_mean*cum_sum) / entry_cnt + cum_mean.pow(2).

then cum_squared_mean should = (cum_pow_sum - 2cum_meancum_sum) / entry_cnt In your code cum_squared_mean=cum_squared_sum/cum_num. as entry_cnt=cum_sum, cum_squared_sum should = cum_pow_sum-2cum_meancum_sum

however cum_squared_sum is defined as torch.cumsum(step_pow_sum, dim=1) which equals cum_pow_sum in the official implementation, so you're missing 2cum_meancum_sum.

am I missing anything here? or is this omitted for a speed / accuracy tradeoff?

$\begin{align*} f&=(f_{n,k})\in\mathbb{R}^{N\times T}...(\mathrm{input}) \end{align*}$ $\begin{align*} s_{k} &=\sum_{n=1}^{N}f_{n,k}...(\mathrm{step\_sum}) \end{align*}$ $\begin{align*} \tilde{s}_{k} &=\sum_{n=1}^{N}f_{n,k}^{2}...(\mathrm{step\_pow\_sum}) \end{align*}$ $\begin{align*} c_{k} &= \sum_{t=1}^{k}s_{t}...(\mathrm{cum\_sum}) \end{align*}$ $\begin{align*} \tilde{c}_{k} &= \sum_{t=1}^{k}\tilde{s}_{t}...(\mathrm{cum\_pow\_sum}) \end{align*}$ $\begin{align*} \mu_{k} &= \frac{1}{Nk}c_{k}...(\mathrm{cum\_mean}) \end{align*}$

In official implementation: $\begin{align*} \sigma_{k}^{2} &= \frac{1}{Nk}(\tilde{c}_{k} - 2\mu_{k}c_{k}) + \mu_{k}^{2} = \frac{1}{Nk}\tilde{c}_{k} - 2\mu_{k}\left(\frac{1}{Nk}c_{k}\right) + \mu_{k}^{2} = \frac{1}{Nk}\tilde{c}_{k} - 2\mu_{k}^{2} + \mu_{k}^{2} = \frac{1}{Nk}\tilde{c}_{k} - \mu_{k}^{2}...(\mathrm{cum\_var}) \end{align*}$

In my repo: $\begin{align*} \tilde{\mu}_{k} &= \frac{1}{Nk}\tilde{c}_{k}...(\mathrm{cum\_pow\_mean}) \end{align*}$ $\begin{align*} \sigma_{k}^{2} &= \tilde{\mu}_{k}-\mu_{k}^{2} = \frac{1}{Nk}\tilde{c}_{k} - \mu_{k}^{2}...(\mathrm{cum\_var}) \end{align*}$

I renamed some variables in check_layer_norm.ipynb at https://github.com/tky823/DNN-based_source_separation/issues/101#issuecomment-975686051 for readability.

My bad! thank you for clearing it up!

tky823 / DNN-based_source_separation

Conv-TasNet Cumulative Layer Norm Bug? #101