search

Rikorose / DeepFilterNet

Noise supression using deep filtering
https://huggingface.co/spaces/hshr/DeepFilterNet2
Other
2.4k stars 223 forks source link

Can't reproduce DeepFilterNet2 and DeepFilterNet3 results on Voicebank Demand test set #537

Closed WoodieDudy closed 2 months ago

WoodieDudy commented 6 months ago

I am currently on last main commit f2445da

I downloaded Voicebank Demand test set from here
I evaluated DeepFilterNet versions 1-3 using the test_voicebank_demand.py script.

Results of DeepFilterNet1 are the same as those reported in the DFN3 paper However, the results of DFN2 and 3 and are much lower.

image

How I can reproduce results similar to those in the paper?

python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet/
2024-03-28 08:47:53 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:47:53 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:47:53 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:47:53 | INFO     | DF | Loading model settings of DeepFilterNet
2024-03-28 08:47:53 | INFO     | DF | Initializing model `deepfilternet`
2024-03-28 08:47:53 | WARNING  | DF | Could not find `best` checkpoint. Checking for default...
2024-03-28 08:47:53 | INFO     | DF | Found checkpoint models/DeepFilterNet/checkpoints/model_30.ckpt with epoch 30
2024-03-28 08:47:53 | INFO     | DF | Running on device cpu
2024-03-28 08:47:53 | INFO     | DF | Model loaded
2024-03-28 08:49:56 | INFO     | DF | Progress: 25%
2024-03-28 08:52:02 | INFO     | DF | Progress: 50%
2024-03-28 08:54:06 | INFO     | DF | Progress: 75%
2024-03-28 08:56:09 | INFO     | DF | Progress: 100%
2024-03-28 08:56:09 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:56:10 | INFO     | DF | Enhanced STOI: 0.9420673982468806
2024-03-28 08:56:10 | INFO     | DF | Enhanced PESQ: 2.809007406234741
2024-03-28 08:56:10 | INFO     | DF | Enhanced CSIG: 4.124786853790283
2024-03-28 08:56:10 | INFO     | DF | Enhanced CBAK: 3.3116347789764404
2024-03-28 08:56:10 | INFO     | DF | Enhanced COVL: 3.472588062286377
2024-03-28 08:56:10 | INFO     | DF | Enhanced SSNR: 7.947866439819336
2024-03-28 08:56:10 | INFO     | DF | Enhanced SISDR: 16.64629923125494
0.9420673982468806,2.809007406234741,4.124786853790283,3.3116347789764404,3.472588062286377,16.64629923125494
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet2/
2024-03-28 08:58:36 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:58:36 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:58:36 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:58:36 | INFO     | DF | Loading model settings of DeepFilterNet2
2024-03-28 08:58:36 | INFO     | DF | Initializing model `deepfilternet2`
2024-03-28 08:58:36 | INFO     | DF | Found checkpoint models/DeepFilterNet2/checkpoints/model_96.ckpt.best with epoch 96
2024-03-28 08:58:36 | INFO     | DF | Running on device cpu
2024-03-28 08:58:36 | INFO     | DF | Model loaded
2024-03-28 08:59:46 | INFO     | DF | Progress: 25%
2024-03-28 09:00:55 | INFO     | DF | Progress: 50%
2024-03-28 09:02:06 | INFO     | DF | Progress: 75%
2024-03-28 09:03:19 | INFO     | DF | Progress: 100%
2024-03-28 09:03:19 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 09:03:20 | INFO     | DF | Enhanced STOI: 0.8134182890036015
2024-03-28 09:03:20 | INFO     | DF | Enhanced PESQ: 2.714287281036377
2024-03-28 09:03:20 | INFO     | DF | Enhanced CSIG: 3.8609461784362793
2024-03-28 09:03:20 | INFO     | DF | Enhanced CBAK: 2.533511161804199
2024-03-28 09:03:20 | INFO     | DF | Enhanced COVL: 3.253587484359741
2024-03-28 09:03:20 | INFO     | DF | Enhanced SSNR: -2.2124998569488525
2024-03-28 09:03:20 | INFO     | DF | Enhanced SISDR: -22.458605475868413
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet3/
2024-03-28 08:40:25 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:40:25 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:40:25 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:40:25 | INFO     | DF | Loading model settings of DeepFilterNet3
2024-03-28 08:40:25 | INFO     | DF | Initializing model `deepfilternet3`
2024-03-28 08:40:25 | INFO     | DF | Found checkpoint models/DeepFilterNet3/checkpoints/model_120.ckpt.best with epoch 120
2024-03-28 08:40:25 | INFO     | DF | Running on device cpu
2024-03-28 08:40:25 | INFO     | DF | Model loaded
2024-03-28 08:41:33 | INFO     | DF | Progress: 25%
2024-03-28 08:42:42 | INFO     | DF | Progress: 50%
2024-03-28 08:43:52 | INFO     | DF | Progress: 75%
2024-03-28 08:45:04 | INFO     | DF | Progress: 100%
2024-03-28 08:45:04 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:45:05 | INFO     | DF | Enhanced STOI: 0.6192451893910225
2024-03-28 08:45:05 | INFO     | DF | Enhanced PESQ: 3.1094107627868652
2024-03-28 08:45:05 | INFO     | DF | Enhanced CSIG: 3.632720708847046
2024-03-28 08:45:05 | INFO     | DF | Enhanced CBAK: 2.6275570392608643
2024-03-28 08:45:05 | INFO     | DF | Enhanced COVL: 3.30583119392395
2024-03-28 08:45:05 | INFO     | DF | Enhanced SSNR: -2.231890916824341
2024-03-28 08:45:05 | INFO     | DF | Enhanced SISDR: -24.651835330047653
0.6192451893910225,3.1094107627868652,3.632720708847046,2.6275570392608643,3.30583119392395,-24.651835330047653

my_requirements.txt

QQQQQQQQY commented 6 months ago

I am currently on last main commit f2445da

I downloaded Voicebank Demand test set from here I evaluated DeepFilterNet versions 1-3 using the test_voicebank_demand.py script.

Results of DeepFilterNet1 are the same as those reported in the DFN3 paper However, the results of DFN2 and 3 and are much lower. image

How I can reproduce results similar to those in the paper?

python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet/
2024-03-28 08:47:53 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:47:53 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:47:53 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:47:53 | INFO     | DF | Loading model settings of DeepFilterNet
2024-03-28 08:47:53 | INFO     | DF | Initializing model `deepfilternet`
2024-03-28 08:47:53 | WARNING  | DF | Could not find `best` checkpoint. Checking for default...
2024-03-28 08:47:53 | INFO     | DF | Found checkpoint models/DeepFilterNet/checkpoints/model_30.ckpt with epoch 30
2024-03-28 08:47:53 | INFO     | DF | Running on device cpu
2024-03-28 08:47:53 | INFO     | DF | Model loaded
2024-03-28 08:49:56 | INFO     | DF | Progress: 25%
2024-03-28 08:52:02 | INFO     | DF | Progress: 50%
2024-03-28 08:54:06 | INFO     | DF | Progress: 75%
2024-03-28 08:56:09 | INFO     | DF | Progress: 100%
2024-03-28 08:56:09 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:56:10 | INFO     | DF | Enhanced STOI: 0.9420673982468806
2024-03-28 08:56:10 | INFO     | DF | Enhanced PESQ: 2.809007406234741
2024-03-28 08:56:10 | INFO     | DF | Enhanced CSIG: 4.124786853790283
2024-03-28 08:56:10 | INFO     | DF | Enhanced CBAK: 3.3116347789764404
2024-03-28 08:56:10 | INFO     | DF | Enhanced COVL: 3.472588062286377
2024-03-28 08:56:10 | INFO     | DF | Enhanced SSNR: 7.947866439819336
2024-03-28 08:56:10 | INFO     | DF | Enhanced SISDR: 16.64629923125494
0.9420673982468806,2.809007406234741,4.124786853790283,3.3116347789764404,3.472588062286377,16.64629923125494
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet2/
2024-03-28 08:58:36 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:58:36 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:58:36 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:58:36 | INFO     | DF | Loading model settings of DeepFilterNet2
2024-03-28 08:58:36 | INFO     | DF | Initializing model `deepfilternet2`
2024-03-28 08:58:36 | INFO     | DF | Found checkpoint models/DeepFilterNet2/checkpoints/model_96.ckpt.best with epoch 96
2024-03-28 08:58:36 | INFO     | DF | Running on device cpu
2024-03-28 08:58:36 | INFO     | DF | Model loaded
2024-03-28 08:59:46 | INFO     | DF | Progress: 25%
2024-03-28 09:00:55 | INFO     | DF | Progress: 50%
2024-03-28 09:02:06 | INFO     | DF | Progress: 75%
2024-03-28 09:03:19 | INFO     | DF | Progress: 100%
2024-03-28 09:03:19 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 09:03:20 | INFO     | DF | Enhanced STOI: 0.8134182890036015
2024-03-28 09:03:20 | INFO     | DF | Enhanced PESQ: 2.714287281036377
2024-03-28 09:03:20 | INFO     | DF | Enhanced CSIG: 3.8609461784362793
2024-03-28 09:03:20 | INFO     | DF | Enhanced CBAK: 2.533511161804199
2024-03-28 09:03:20 | INFO     | DF | Enhanced COVL: 3.253587484359741
2024-03-28 09:03:20 | INFO     | DF | Enhanced SSNR: -2.2124998569488525
2024-03-28 09:03:20 | INFO     | DF | Enhanced SISDR: -22.458605475868413
python DeepFilterNet/df/scripts/test_voicebank_demand.py /datasets/testset2 --model-base-dir models/DeepFilterNet3/
2024-03-28 08:40:25 | INFO     | DF | Running on torch 1.13.1+cpu
2024-03-28 08:40:25 | INFO     | DF | Running on host sbx-4c5addaafaa7409c8eec87d5eb109b90-5968599-gstbl
2024-03-28 08:40:25 | INFO     | DF | Git commit: 1097015, branch: torchDF-speedup-stft
2024-03-28 08:40:25 | INFO     | DF | Loading model settings of DeepFilterNet3
2024-03-28 08:40:25 | INFO     | DF | Initializing model `deepfilternet3`
2024-03-28 08:40:25 | INFO     | DF | Found checkpoint models/DeepFilterNet3/checkpoints/model_120.ckpt.best with epoch 120
2024-03-28 08:40:25 | INFO     | DF | Running on device cpu
2024-03-28 08:40:25 | INFO     | DF | Model loaded
2024-03-28 08:41:33 | INFO     | DF | Progress: 25%
2024-03-28 08:42:42 | INFO     | DF | Progress: 50%
2024-03-28 08:43:52 | INFO     | DF | Progress: 75%
2024-03-28 08:45:04 | INFO     | DF | Progress: 100%
2024-03-28 08:45:04 | INFO     | DF | Waiting for metrics computation completion. This could take a few minutes.
2024-03-28 08:45:05 | INFO     | DF | Enhanced STOI: 0.6192451893910225
2024-03-28 08:45:05 | INFO     | DF | Enhanced PESQ: 3.1094107627868652
2024-03-28 08:45:05 | INFO     | DF | Enhanced CSIG: 3.632720708847046
2024-03-28 08:45:05 | INFO     | DF | Enhanced CBAK: 2.6275570392608643
2024-03-28 08:45:05 | INFO     | DF | Enhanced COVL: 3.30583119392395
2024-03-28 08:45:05 | INFO     | DF | Enhanced SSNR: -2.231890916824341
2024-03-28 08:45:05 | INFO     | DF | Enhanced SISDR: -24.651835330047653
0.6192451893910225,3.1094107627868652,3.632720708847046,2.6275570392608643,3.30583119392395,-24.651835330047653

my_requirements.txt

That's what I'm getting. Did you manage to fix it?

WoodieDudy commented 6 months ago

I have updated torch to 2.2 and that fixed problem. There's is my new requirements reqs.txt But I haven't figured out what exactly was causing the problem yet. I'll text when I find out.

QQQQQQQQY commented 6 months ago

我已将火炬更新到 2.2 并解决了这个问题。这是我的新要求reqs.txt但我还没有弄清楚到底是什么导致了这个问题。当我发现时，我会发短信。

I have updated torch to 2.2 and that fixed problem. There's is my new requirements reqs.txt But I haven't figured out what exactly was causing the problem yet. I'll text when I find out.

I appreciate you writing back. I'll try it.

QQQQQQQQY commented 6 months ago

我已将火炬更新到 2.2 并解决了这个问题。这是我的新要求reqs.txt但我还没有弄清楚到底是什么导致了这个问题。当我发现时，我会发短信。

I have updated torch to 2.2 and that fixed problem. There's is my new requirements reqs.txt But I haven't figured out what exactly was causing the problem yet. I'll text when I find out. Hi, I upgraded torch=2.2.2. the last result of Deepfilter2 is as follows: 2024-04-04 16:45:36 | INFO | DF | Running on torch 2.2.2+cu118 2024-04-04 16:45:36 | INFO | DF | Running on host DESKTOP-J0UR71S 2024-04-04 16:45:36 | INFO | DF | Loading model settings of outputs 2024-04-04 16:45:36 | INFO | DF | Initializing model deepfilternet2 2024-04-04 16:45:38 | INFO | DF | Found checkpoint D:\DeepFilterNet-main\DeepFilterNet\outputs\checkpoints\model_60.ckpt.best with epoch 60 2024-04-04 16:45:38 | INFO | DF | Running on device cuda:0 2024-04-04 16:45:38 | INFO | DF | Model loaded 2024-04-04 16:45:53 | INFO | DF | Progress: 25% 2024-04-04 16:46:08 | INFO | DF | Progress: 50% 2024-04-04 16:46:22 | INFO | DF | Progress: 75% 2024-04-04 16:46:36 | INFO | DF | Progress: 100% 2024-04-04 16:46:36 | INFO | DF | Waiting for metrics computation completion. This could take a few minutes. 2024-04-04 16:48:02 | INFO | DF | Enhanced STOI: 0.925880108458049 2024-04-04 16:48:02 | INFO | DF | Enhanced PESQ: 2.823953866958618 2024-04-04 16:48:02 | INFO | DF | Enhanced CSIG: 4.110456466674805 2024-04-04 16:48:02 | INFO | DF | Enhanced CBAK: 3.1988513469696045 2024-04-04 16:48:02 | INFO | DF | Enhanced COVL: 3.4739575386047363 2024-04-04 16:48:02 | INFO | DF | Enhanced SSNR: 6.198051452636719 2024-04-04 16:48:02 | INFO | DF | Enhanced SISDR: 13.75959416670562 It still doesn't achieve the results in the original article, so may I ask if you succeeded? I found your best epoch at 96 while I'm at 60, can you share your config.ini? I'm very much looking forward to your reply

WoodieDudy commented 6 months ago

I took weights and config.ini from there

satvik-venkatesh commented 5 months ago

@WoodieDudy upgrading torch to 2.2 surprisingly worked!! I also installed pystoi and pesq package versions from your requirements file. Thank you.

github-actions[bot] commented 2 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.