I am running the code in a dual NVIDIA RTX 3090 environment, but the time to train one epoch has increased from around 2 hours to now over 10 hours, and it still hasn't finished training one epoch. Could you please tell me why this is happening?
I tried on the official code and ran successfully. It takes 26 minutes to train one epoch with 4 3090 GPUs, and 48 minutes with 2 3090 GPUs. You may check whether your data is correctly organized.
I am running the code in a dual NVIDIA RTX 3090 environment, but the time to train one epoch has increased from around 2 hours to now over 10 hours, and it still hasn't finished training one epoch. Could you please tell me why this is happening?