-
### Feature request
The current implementation of the LLAMA model in the Hugging Face Transformers repository supports self-attention layers as per the standard design of transformer models. I prop…
-
The Cross-Layer Attention (CLA) proposed by MIT recently can significantly reduce runtime memory usage. Does vLLM have any plans to support it? Thanks!
Cross-Layer Attention paper: https://arxiv.or…
-
I am interested in performing multimodal cross-attention. I don't see issues in performing self-attention in encoder since i can use the `BertAttention` plugin. However, cross-attention would have `qu…
-
when I run "python src/eval.py --dataset_path --batch_size --mixed_precision fp16 --output_dir --save_name --num_workers_test --sketch_cond_rate 0.2 --dataset --start_cond_rate 0.0 --test_order …
-
For the same training settings on both trainer:
512x512
Adamw8bit
4 batch size
FluxGym : 1.19s/it
ComfyUI FluxTrainer: 7.31s/it
About 6 times faster on FluxGym, on FluxTrainer my gpu utilisa…
-
Hi, @lucidrains. Thank you for sharing this excellent implementation with us all!
Do you have any thoughts as to what changes would need to be made to make cross-attention possible with your `FLASH` …
-
Hi @fradif96
We don't have new modules for cross/self-attention. It's the same attention layers but just reshape the latent features from ((b t) l d) -> (b (t l) d) [here](https://gith…
-
Thank you for your implementation of SAMI's training code. It has been incredibly helpful for me!
However,I have a question regarding the pretraining process. In forward preprocess of the [MaeDeco…
-
Dear Authors,
Thank you very much for the open source code.
I have a cross attention module for which I would like to use flash attention. But unfortunately, I get the following error:
`*** …
-
Boss? Is there an example of CLIP+U-Net implementing cross attention? / 大老? 有CLIP+U-Net实现交叉注意cross-attention的例子吗?