search

haoshao-nku / medical_seg

181 stars 21 forks source link

About training time #8

Closed lxr-1204 closed 5 months ago

lxr-1204 commented 5 months ago

Thank you very much for your excellent work. I followed the instructions in the README to run Polyper, but I found that the training time takes nearly 3 days. Additionally, I noticed that the GPU utilization and GPU memory usage are as shown in the graph below, with relatively low GPU utilization. Is this situation normal? GPU is NVIDIA GeForce RTX 3080, 20G

image image

If possible, could you please share your log files? I would greatly appreciate it.

Thank you!

haoshao-nku commented 5 months ago

Thanks for your attention. I observe that you spend too much time on each iter. We used the single 3090 to train a total of 8000 iter, which took no more than 24h, and the time for each iter was about 1s.

lxr-1204 commented 5 months ago

Thank you for your so prompt response! I have switched to a machine with RTX 3090, and as you said, it will finish running within a day. Additionally, I would like to ask about the learning rate of 0.0002, a momentum of 0.9, and a weight decay of 1e-4 mentioned in the Implementation Details of the paper and in local_config/base/schedules/schedule_80k_adamw.py. However, you made some changes in the local_config/Polyper-AAAI2024/main/polyper_polypseg_224*224_80k.py file as shown in the figure. Could you please advise me on which one to follow for optimal practical results?

image

Once again, I appreciate your assistance.

haoshao-nku commented 5 months ago

Please follow the experimental setup described in our paper.