search

tsurumeso / vocal-remover

Vocal Remover using Deep Neural Networks
MIT License
1.55k stars 222 forks source link

Unrealistic amounts of RAM required for training #3

Closed AndroXD closed 4 years ago

AndroXD commented 4 years ago

Hello, I'm trying to train on a dataset with thousands of .wav files that are about 4 minutes long, but I run out of RAM with anything larger than 75 files with 16GB of RAM:

Traceback (most recent call last):
  File "train.py", line 98, in <module>
    hop_length=args.hop_length)
  File "C:\vocal-remover\lib\dataset.py", line 33, in create_dataset
    (len_dataset, 2, hop_length, cropsize), dtype=np.float32)
MemoryError

So theorically I would still run out of RAM even if I had 128GB and tried to train on more than 600 files (128 / 16 = 8 , 8 * 75 = 600) , I'm trying to achieve similar results to http://krisp.ai , but that needs thousands of files, not just a few hundred. Is it possible to increase the number of files or make the training scalable? (ie. split it on 75 files chunks and resume training on the next chunk automatically)

tsurumeso commented 4 years ago

Yeah, it is possible to make the training scalable by dynamic data loading. However, since I'm busy recently, I think it will take some time to implement it. Sorry for your inconvenience. FYI, the pre-trained model on this repository was trained with 174 files with 32GB of RAM.