search

smu-ivpl / FastInterCU-VVC

Official implementation of "Deep Neural Network-based Fast Inter CU Splitting Algorithm for Versatile Video Coding (VVC) Standard"
29 stars 11 forks source link

Why is the training process on my computer so slow? #2

Closed tengqi200 closed 1 year ago

tengqi200 commented 2 years ago

The whole training process takes 64 days on my computer, and I didn't modify the code provided by the author. The software and hardware configuration are as follows: --Inter(R) Xeon(R) W-21236 CPU @ 3.60GHz --RAM 16.0GB --Mechanical hard disk --GTX 1080Ti 11.0GB --Windows 10 --pytorch 1.10.0 --cuda 11.3

unacanal commented 2 years ago

One of the reason may be the GPU performance. I used two V100 for training. But I'm not sure it's the main problem.

Maybe you should try changing 'num_worker_per_gpu: 4' to '1'. I have experienced slow training issue in another project when setting num_worker value as greater than 1.

tengqi200 commented 2 years ago

One of the reason may be the GPU performance. I used two V100 for training. But I'm not sure it's the main problem.

Maybe you should try changing 'num_worker_per_gpu: 4' to '1'. I have experienced slow training issue in another project when setting num_worker value as greater than 1.

I found that the utilization rate of the GPU is 0%. I think the data loading process is the bottleneck. So I tried some ways to solve the issue. 

If I train the model in Ubuntu and change the "num_worker_per_gpu" from 4 to 8, the total time of the training process can be reduced from 64 days to 30 days. If the "num_worker_per_gpu" is set to 1, the total time can be reduced to 20 days. However, the utilization rate of the GPU is still low (occasionally up to 44%). Moving data sets to SSD should speed up training.

On the other hand, the loss of this model is reduced to 0.3 before 1000 iterations, and then it hardly becomes lower, and the accuracy can reach 80%. So, is it too large to set the total iteration to be 600k (the default setting)? 

Anyway, I think your work is interesting and inspiring.