Hi
When I start training, time of train of any step is good and it uses GPU but after 32 steps (if _batches_per_group=32 in datafeeder) GPU utilization going to 0 and after many seconds the queue of data becomes ready and again training starts.
I saw datafeeder.py and it uses threading for queue of data but why it stops training? How can I increase GPU utilization?
I increased _batches_per_group in datafeeder.py but it causes training stops more seconds. Following picture show that if I set _batches_per_group=128 training stops for 89 seconds!
Hi When I start training, time of train of any step is good and it uses GPU but after 32 steps (if
_batches_per_group=32in datafeeder) GPU utilization going to 0 and after many seconds the queue of data becomes ready and again training starts. I saw datafeeder.py and it uses threading for queue of data but why it stops training? How can I increase GPU utilization? I increased_batches_per_groupin datafeeder.py but it causes training stops more seconds. Following picture show that if I set_batches_per_group=128training stops for 89 seconds!