[Discussion] Stop training and resume training

[ByteMe666]

Hello, I'm a noob to Spleeter and AI, and am completely going like the Wizard's apprentice here. I've created a dataset and was excited to see Spleeter starting training. However it was very long and my laptop's fan would sound like an airplane taking off, the computer heated up quite a bit, and after hours of running there was no end in sight. So, I figured I didn't mind waiting long, but I didn't like my computer heating up. I'm not interested enough to buy a GPU and the non-laptop computer that goes with it to speed up training, so I elected to have a computer on the cloud do the work for me, while I carry on with my life. However the cloud job has a 1 hour time limit, so I used the caching feature of the online cloud service to cache both my cache folder and model folder between each run: I was hoping every time a new job starts, Spleeter would have the cache and model folders in the state when the last job quit after 1 hour. It seems to be working, I see a new job pop up every hour and I see a message: "INFO:tensorflow:Restoring parameters from /myaudiodatabase/myaudiodatabase_model/model.ckpt-0"

Besides that I see a lot of "audio loaded successfully", but only occasionally something about tensorflow, even though I'm using the --verbose option

In case the computation slowed down, I reduced save_checkpoints_steps from 1000 to 100 to get more checkpoints. Is that a good idea ?

Is my approach actually leading to Spleeter eventually finishing, or is my project restarting every hour from the same point ? I'm not 100% sure what I'm doing is actually working. How can I tell when Spleeter is almost finished training ?

Thank you very much.

[ByteMe666]

Hi again, So it turns out the caching features was immutable and only the first cache from the first run was used over and over. I found a workaround and now I see model.ckpt-100, model.ckpt-200 etc. get used during each iteration. It seems to be working. Unfortunately the cloud service has a limited amount of build time, so I can't use it to build for longer than I could at home. The cloud service I use can offer me a VM with GPU to train Spleeter, but I have to pay for it. It's not very expensive I think but there is a lot of paperwork overhead, private data I must giveaway, and a bunch of automatic payment subscriptions that have to be setup before I can even try it out, and it annoys me enough to think it's not worth the hassle. Nice to know though that you can get a GPU VM online to run the computation and not have to invest in hardware just to train Spleeter. If I ever reach a point where I have enough need for a GPU machine and am willing to spend the money and do all the private data giveaway, I might bite the bullet and do it. For now my little Spleeter experiments are not enough to justify subscribing to a cloud service with automatic payments.

[hassan8971]

Hi ByteMe666, I'm also new to ai and spleeter, but my question is how to even create a dataset for spleeter and i would appreciate it if you help me in this case!

[ByteMe666]

Hi, That part is rather well explained in the wiki: https://github.com/deezer/spleeter/wiki/2.-Getting-started

Look at the links to the .json file examples.

You need a json file that explains where to find the training data, a json file that explains where to find the validation data (about 10% amount of data compared to the training data), and of course: the data.

.wav files are fine as data, you need a .wav file with an example of a song, and two .wav files where the voice and the instruments are isolated. You need a bunch of songs like that. About 100 of them (the default Spleeter training data set has 87).

Then just run the training command from the documentation.

Where you get that training data: that's up to you to find. You can look for it on Google. I found a website called Kaggle with a bunch of datasets, but you have to go fishing and hope to find a good dataset.

If you do not find a dataset like MusDB where all the files are provided for you, you can create data, here are some things you can try, but your ability to execute them may depend on how familiar you are with music software and programs:

1) You could find an instrumental music dataset, and an a capella singing dataset and combine them to create .wav files where voices and instruments are mixed. For that you could use a script and the program ffmpeg (that you run in a terminal), here is the basic command: ffmpeg -i instrumental_1.wav -i vocals_1.wav -filter_complex '[0][1]amix,volume=2.0' mix_1.wav

Note that the volume is doubled because the "amix" audio effect divides the volume by two by default, which is not desired. You would need to run that command over and over in a script and generate mix2.wav, mix_3.wav etc. Be careful that you use and generate .wav files with matching sample rates, bit depth etc. ffmpeg can convert everything for you but I leave that up to you to figure out the command, ffmpeg is well documented on the internet.

I wonder how well that approach would work: the singing will be out of key, and with a completely different rhythm, but maybe Spleeter does not care about that. Also the singing would start immediately at the beginning of the song, maybe you shouldn't train Spleeter to expect singing to be always right at the beginning, so you can delay when the singing starts (again, something ffmpeg can achieve for you, but you need to look up some commands by yourself)

2) You could buy music software (Reason, Ableton, Native Instruments Samplers (...)) and generate 100 instrumental tracks by using music loops and their features to keep the music in the correct key and rhythm. You could then mix that with your a capella vocals. It would be some work to do this for 100 tracks, but it's not so much work that it's impossible to do. It might eat up a couple of weekends if you are already familiar with music making software. If you've never used music software before: you'll need to learn to use music software first.

Hope this helps.

[hassan8971]

Thank you so much, great explanation!

[hassan8971]

I also need to find the original model files! (jsons, csvs and anything else which is related to this) I actually wanna run the spleeter with python code, but I get errors when using the sample codes which spleeter has put on its github page. Do you have any sample python code to do this which works?

[renjunok]

How to generate a new model based on 2stems?

hope to get your reply

Thank you!