how to increase gpu usage?

[ArtemisZGL]

i use 6 gpu to train the model with batch size 200, but each gpu-util is only between 0-22%, and cost 2 hours per epoch, but memory usage is high. The time is not increased compared with 2 gpu, while 2 gpu-util is about 80%.

and then i change the multi-thread to 128, the usage can be increased to 30-50%, and cost 1 hour per epoch, is there any way to increased the gpu usage anymore?

[WeidiXie]

Hi, I think the code is bottlenecked by the data loading part, because of the librosa.

If you change that part to tensorflow, it will cut your time dramatically.

[ArtemisZGL]

@WeidiXie hi, thanks for your reply. I also want to know why i can't set the fit_generator use_multiprocessing to True, and set workers to n, or what is the different between those two methods using multiprocessing?

[WeidiXie]

Oh, I just don't get used to using the default multiprocessing in Keras. So I never use it.

[ArtemisZGL]

@WeidiXie thanks, but if i set the fit_generator use_multiprocessing to True, the program didn't work anymore, it will just be froze after log Epoch 1/128 Learning rate for epoch 1 is 0.0001.

is there any problem when using both these two methods?(it can works when i only one of then alone.)

[WeidiXie]

Well, I guess then you can't use two multiprocessing for loading data, either use their default one, or use the one in my code.

[ArtemisZGL]

ok, thanks a lot!