May I ask about the training time for retrieval task?
I noticed that the code is using 6 GPUs, while I am currently using 4*RTX4090 and it went fast on the 30 epochs of base model but seems very slow on the following 300 epochs. May I know about the total training time on your side?
@RyanG41 I think it is normal. The following 300 epochs train a much large Transformer model than previous 30 epochs. I think it takes about 20 hours on 6 2080Ti GPUs if I remember correctly.
Hi,
May I ask about the training time for retrieval task?
I noticed that the code is using 6 GPUs, while I am currently using 4*RTX4090 and it went fast on the 30 epochs of base model but seems very slow on the following 300 epochs. May I know about the total training time on your side?
Thanks a lot.
Best Regard