Closed AndreaPi closed 4 months ago
Very interesting work! I see you pinned torch==2.1.2 . PyTorch 2.2 promises a 2x improvement to scaled_dot_product_attention and a few torch.compile improvements:
torch==2.1.2
scaled_dot_product_attention
torch.compile
https://pytorch.org/blog/pytorch2-2/
I don't think using PyTorch 2.2 will result in a big speed increase for GPTFast, but still it may be nice to try. What do you think?
I definitely agree! Apologies for the late response-I will probably upgrade it in the next version, which will be out soon.
Hey, the new version does indeed have torch==2.2! Closing this now :)
Very interesting work! I see you pinned
torch==2.1.2
. PyTorch 2.2 promises a 2x improvement toscaled_dot_product_attention
and a fewtorch.compile
improvements:https://pytorch.org/blog/pytorch2-2/
I don't think using PyTorch 2.2 will result in a big speed increase for GPTFast, but still it may be nice to try. What do you think?