First of all, thank you for this fantastic paper. Tempeh helps me a lot with my studies. However, I noticed that the paper says during the course stage, tempeh trained on 300K iterations. Let’s say 300K took a week to train. For each iteration, tempeh only took less than one second to train. However, I used PyTorch to rewrite your code using a similar architecture. It took me 50 seconds for one iteration only on 2000 data. I am using 2 a100 GPUs. I tried to figure it out. Is there anything Tempeh did to speed it up to this incredible speed? It would be very nice of you to share any tips on how to speed up PyTorch training.
AdamLee19
commented
2 weeks ago
First of all, thank you for this fantastic paper. Tempeh helps me a lot with my studies. However, I noticed that the paper says during the course stage, tempeh trained on 300K iterations. Let’s say 300K took a week to train. For each iteration, tempeh only took less than one second to train. However, I used PyTorch to rewrite your code using a similar architecture. It took me 50 seconds for one iteration only on 2000 data. I am using 2 a100 GPUs. I tried to figure it out. Is there anything Tempeh did to speed it up to this incredible speed? It would be very nice of you to share any tips on how to speed up PyTorch training.