huyanxin / DeepComplexCRN

Apache License 2.0
394 stars 97 forks source link

Question about benchmarks in paper #2

Closed jonashaag closed 4 years ago

jonashaag commented 4 years ago

Hi! Thanks for the paper, it’s a nice read, and congratulations for performing so well in the DNS challenge!

I have a question about the training procedure used for the models that are being compared in the paper. Did you train all of the models (except the E-Aug variant) on identical data and with same number of epochs, augmentation, etc? In particular, did you use the same training data and process for the CL variant and DCUNet-16 or did you use a version of DCUNet trained on a different dataset? Specifically, did you use RIRs when training DCUNet?

I am asking because I wonder if there is anything inherent to DCUNet that makes it worse for reverberated speech or if the 0.5 PESQ difference is a result of not training on reverberated data.

Did you compare to any other DCUNet variant than DCUNet-16?

Thank you!

huyanxin commented 4 years ago

Thank you for your concern about my work!

Due to the limitation of computing resources, an early stop strategy was used in our training procedure. Although the epochs were not exactly the same they are similar. Beyond that, the configurations ware the same when we trained models, include random seeds. We also use our private dataset on those models, the conclusion is similar.

It's a good problem with DCUNet-16. For DNS, I make some changes on it about the strides to make causal or semi-causal when doing streaming. The targets of all models were the clean wave with reverberant. So, we speculated the context information influenced the results. And we add the dilation in DCUNet-16. But it was not worked. Maybe the reverberated data was not the core problem. The noise of some waves in DNS's realrec testset can not be suppressed (some waves' tag is audioset aircondition). At last, DCCRN was our final plan for DNS.

As for DCUNet-20 or DCUNet-10, they need more changes to be a fair comparison. Considered the more changes we make, the farther away our solution is from the original. So we did not compare others on an open-source set. Thank you for your attention!

jonashaag commented 4 years ago

Interesting, thank you!