search

sigsep / open-unmix-pytorch

Open-Unmix - Music Source Separation for PyTorch
https://sigsep.github.io/open-unmix/
MIT License
1.24k stars 181 forks source link

Hardware requirements for test #7

Closed hagenw closed 4 years ago

hagenw commented 5 years ago

🐛 Bug

It seems to be relatively easy to get out of memory for the first example provided in the README on the GPU. Maybe it would be nice to add some hardware requirements or estimation how much memory you need per second of input signal.

To Reproduce

Steps to reproduce the behavior:

>>> python test.py ~/data/musdb18-wav/test/Al\ James\ -\ Schoolboy\ Facination/mixture.wav --model umxhq
Traceback (most recent call last):
  File "test.py", line 301, in <module>
    device=device
  File "test.py", line 166, in separate
    use_softmask=softmask)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-gpu/lib/python3.7/site-packages/norbert/__init__.py", line 260, in wiener
    y = expectation_maximization(y/max_abs, x_scaled, iterations, eps=eps)[0]
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-gpu/lib/python3.7/site-packages/norbert/__init__.py", line 141, in expectation_maximization
    eps)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-gpu/lib/python3.7/site-packages/norbert/__init__.py", line 511, in get_local_gaussian_model
    C_j = _covariance(y_j)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-gpu/lib/python3.7/site-packages/norbert/__init__.py", line 468, in _covariance
    y_j.dtype)
MemoryError

Environment

Please add some information about your environment

If unsure you can paste the output from the pytorch environment collection script (or fill out the checklist below manually).

You can get that script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce GTX 1050
Nvidia driver version: 430.40
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.13.3
[conda] mkl                       2019.4                      243  
[conda] pytorch                   1.2.0           py3.7_cuda10.0.130_cudnn7.6.2_0    pytorch
faroit commented 5 years ago

Are you sure that it's hitting the memory error on the GPU? To me, it looks like it is Norbert (meaning the wiener filter on CPU). Can you try running the same example with --niter 0 on the same machine?

hagenw commented 5 years ago

You are right, I get the same error on the CPU:

>>> python test.py ~/data/musdb18/musdb18-wav/test/Al\ James\ -\ Schoolboy\ Facination/mixture.wav --model umx --no-cuda
Traceback (most recent call last):
  File "test.py", line 301, in <module>
    device=device
  File "test.py", line 166, in separate
    use_softmask=softmask)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-linux-cpu/lib/python3.7/site-packages/norbert/__init__.py", line 260, in wiener
    y = expectation_maximization(y/max_abs, x_scaled, iterations, eps=eps)[0]
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-linux-cpu/lib/python3.7/site-packages/norbert/__init__.py", line 141, in expectation_maximization
    eps)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-linux-cpu/lib/python3.7/site-packages/norbert/__init__.py", line 511, in get_local_gaussian_model
    C_j = _covariance(y_j)
  File "/home/audeering.local/hwierstorf/.anaconda3/envs/open-unmix-pytorch-linux-cpu/lib/python3.7/site-packages/norbert/__init__.py", line 468, in _covariance
    y_j.dtype)
MemoryError

I tested with --niter 0 and it worked, thanks.

I have 16GB of RAM which seems to be reasonable, so maybe you add the --niter 0 hint to the first example as well?

faroit commented 5 years ago

I have 16GB of RAM which seems to be reasonable,

Interestingly, .I tried to run the same example on my laptop (OSX, 8GB of RAM) and it worked fine (with some swapping).

so maybe you add the --niter 0 hint to the first example as well?

in any case I will add a note in the readme. We also plan to improve the RAM usage of norbert soon, maybe you are interested in helping out?

hagenw commented 5 years ago

I have to admit, I'm using 6GB by other programs, so it's not really 16GB available here ;)

I would love to work a little bit more on source separation at the moment, but I'm afraid I will not really have time this year.

Feel free to close this issue, after adding a comment to the README.

aadibajpai commented 5 years ago

Can confirm, I have a server with 16GB ram out of which 14GB was available when executing. It was always getting killed when I executed just test.py but worked with --niter=0.

I was wondering if that significantly affects the quality of the output? If not then maybe it could be made the default?

(open-unmix-pytorch-linux-cpu) aadibajpai@ubuntu_hydralatest:~/open-unmix-pytorch$ python test.py photograph.wav 
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [01:06<00:00, 16.82s/it]
Killed
(open-unmix-pytorch-linux-cpu) aadibajpai@ubuntu_hydralatest:~/open-unmix-pytorch$ python test.py photograph.wav --niter=0
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [01:08<00:00, 17.06s/it]
faroit commented 5 years ago

I was wondering if that significantly affects the quality of the output?

yes. The difference on MUSDB18 is about 0.3 dB SDR (vocals), which is significant, so we want keep the wiener filter in the post-processing by default.

If not then maybe it could be made the default?

I will add a comment in the docs/readme as proposed by @hagenw. Also I think it makes sense to add a start and duration argument to the separation function, allowing to crop the input, thus saving memory.

aadibajpai commented 5 years ago

I was wondering if that significantly affects the quality of the output?

yes. The difference on MUSDB18 is about 0.3 dB SDR (vocals), which is significant, so we want keep the wiener filter in the post-processing by default.

That's a lot, in that case do you know why it fails on decent specs? I'm very impressed by the results with niter=0 so can only imagine how well it would work with it. Also, when talking of SDR/SAR/SIR values, higher is better, right?

If not then maybe it could be made the default?

I will add a comment in the docs/readme as proposed by @hagenw. Also I think it makes sense to add a start and duration argument to the separation function, allowing to crop the input, thus saving memory.

That's fair enough but maybe not very useful when you need to work with full length tracks (such as in my case)

aliutkus commented 5 years ago

I modified the norbert package through this pull request

this should make the memory usage smaller @hagenw @aadibajpai could you try it again with the new norbert, through:

pip install norbert==0.2.1
aadibajpai commented 5 years ago

I modified the norbert package through this pull request

this should make the memory usage smaller @hagenw @aadibajpai could you try it again with the new norbert, through:

pip install norbert==0.2.1

worked!

faroit commented 5 years ago

@hagenw did you had time to test this on your machine?

hagenw commented 5 years ago

It's working for me now.

But I agree that it would be nice to have something like start and duration arguments. Or maybe the possibility in the long run to integrate some block-processing internally.

faroit commented 5 years ago

But I agree that it would be nice to have something like start and duration arguments

@hagenw have a look at #10 and feel free to review that one.

Or maybe the possibility in the long run to integrate some block-processing internally.

For bidirectional models that would reduce the performance significantly. Do you think an overlap-add would be sufficient here for test time?

In any case, the unidirectional model would not suffer from that.

faroit commented 4 years ago

But I agree that it would be nice to have something like start and duration arguments. Or maybe the possibility in the long run to integrate some block-processing internally.

start and duration have been merged in #10

@hagenw can this be closed?

faroit commented 4 years ago

@hagenw feel free to reopen if you have further comments