Closed Xingyyy01 closed 1 year ago
What do you mean by "the time of per epoch multiplies"? If you don't mind, you can give me the exact time that you are seeing, and the server setup (e.g., what GPU you are using)
Thanks for your reply, i use NVIDIA TITAN Xp. The time of per epoch multiplies: when I run the main.py in example_training for the first time, it takes about 2 minutes per epoch, when I run the file for the second time, it takes about 4 minutes per epoch, when I run it for the third time, it takes about 14 minutes per epoch, and when I run it for the last time, it takes 40 minutes per epoch. Attachments are files produced by running code. time1.txt time2.txt time3.txt
How many concurrent training jobs are you running on the GPU? you can use nvidia-smi
to see them. If you are running multiple jobs are the same time, the training time would increase
ooh! I see, thank you! I haved reconfigured the environment these days, and it can run normally, it takes about 6 minutes for one epoch Here is the work.log file generated by the current run, could you help me check it? worker.log
It looks normal to me!
great! tysm!
hii, first of all, thanks for your great work! I was able to run the example codes, but i got a problem that as I run the code multiple times, the time of per epoch multiplies. Maybe I didn't understand the code enough, trying to get some help.