Flash Attention 3 3090 support

Dao-AILab / flash-attention

Fast and memory-efficient exact attention

BSD 3-Clause "New" or "Revised" License

13.76k stars 1.26k forks source link

Open win10ogod opened 3 months ago

win10ogod commented 3 months ago

Will there be 3090 support on Flash Attention 3 in the future?

Boubou78000 commented 3 months ago

And 4070?

Boubou78000 commented 3 months ago

=Ada Lovelace.

NVIDIA says that it took the fp8 of the ada architecture from the hopper architecture.

Vhallo commented 3 months ago

3090 support would be surprising, but Blackwell might be another story.

Boubou78000 commented 3 months ago

3090 support would be surprising, but Blackwell might be another story.

We need first to wait it to be deployed by NVIDIA

samsja commented 2 months ago

Will there be 3090 support on Flash Attention 3 in the future?

Flash attention 3 is only working on hopper so only h100 (maybe 4090 ?). 3090 is ampere so no fa3 support :cry:

thakkarV commented 2 months ago

FA2 is already performant on Ampere class chips. FA3 is hardware specific to Hopper