-
### 🚀 The feature, motivation and pitch
FlexAttention was proposed as a performant attention implementation leveraging `torch.compile` with easy APIs for adding support for complex attention varian…
mgoin updated
1 month ago
-
Hi! Thank you for your great work. I was looking at the code and I see that deformable attention is only used in the cross-attention Decoder module.
Why is deformable attention not used anywhere e…
-
Under the 128k long sequence, the activation value memory increases significantly.
CP8 + TP8 seems necessary (they reduce the activation value memory almost linearly), but there is still as much as …
-
Hi,
Today, when I was running LoRA training for the `Flux.1` model (sd-scripts on SD3's breach), the "`train_blocks must be single for split mode`" error suddenly occurred. This error had not appea…
-
### 🚀 The feature, motivation and pitch
Enable support for Flash Attention Memory Efficient and SDPA kernels for AMD GPUs.
At present using these gives below warning with latest nightlies (torch=…
-
Thank you very much for your excellent work. I am now encountering this problem while training my model in a virtual environment. When I execute the command line, an error occurs. Can anyone solve it…
-
~/# accelerate launch train_stage_2.py --config configs/train/stage2.yaml
The following values were not passed to `accelerate launch` and had defaults used instead:
`--num_processes` was set…
-
Hi,
I saw that the demo project has a search filter in the opac, I thought it was really cool but I would like to know if this is new, exclusive to this version, or if I can do it with previous ver…
-
D:\DiffSynth-Studio-main\diffsynth\models\attention.py:54: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:4…
-
### Description
This package looks like a great step up from `piexifjs` in many ways. Better API, supports more image formats (awesome!). But unlike `piexifjs` it does not support writing the Exif …