Update to use torch.nn.attention.sdpa_kernel

pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

BSD 3-Clause "New" or "Revised" License

5.35k stars 484 forks source link

Open yanboliang opened 3 months ago

Chillee commented 3 months ago

The main reason I didn't do this previously is worry that this'll cause the code to hard break on older versions. When was this new API added?

yanboliang commented 3 months ago

It added two months ago (https://github.com/pytorch/pytorch/commit/4e29f01bf2edfe40d6fb8b38228199185645b6af). I understand the concern, don't have strong opinion on changing this.