search

PaddlePaddle / PaddleMIX

Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.
Apache License 2.0
292 stars 110 forks source link

PaddleMIX ppdiffusers中升级attention相关代码 #262

Closed chenjjcccc closed 8 months ago

chenjjcccc commented 10 months ago

PaddleMIX ppdiffusers中升级attention相关代码

任务描述

任务背景

题目更新：

  1. 对比ppdiffusers (v0.19.3)和diffusers (v0.21.1)的代码差异，可使用页面diff工具

当前还请按照diffusers最新稳定版本0.23.1更新升级

unicornshell commented 9 months ago

ppdiffusers里attention.py里GEGLU模块缺少 def gelu(self, gate): if gate.device.type != "mps": return F.gelu(gate)，但这并不在代码差异中，需要更改吗

unicornshell commented 9 months ago

似乎缺少models/attention.py的单测

LokeZhou commented 9 months ago

ppdiffusers里attention.py里GEGLU模块缺少 def gelu(self, gate): if gate.device.type != "mps": return F.gelu(gate)，但这并不在代码差异中，需要更改吗

不需要

LokeZhou commented 9 months ago

似乎缺少models/attention.py的单测

如果torch 的diffusers里有这个单测则添加上去，如果没有就不需要添加

unicornshell commented 9 months ago

diffusers的attention_processor.py里存在的pytorch2.0版本的类需要在ppdiffusers里实现吗 像： AttnAddedKVProcessor2_0，AttnProcessor2_0，LoRAAttnProcessor2_0

unicornshell commented 9 months ago

缺少2_0的类会导致新增的

ADDED_KV_ATTENTION_PROCESSORS = (
    AttnAddedKVProcessor,
    SlicedAttnAddedKVProcessor,
    AttnAddedKVProcessor2_0,
    XFormersAttnAddedKVProcessor,
    LoRAAttnAddedKVProcessor,
)

CROSS_ATTENTION_PROCESSORS = (
    AttnProcessor,
    AttnProcessor2_0,
    XFormersAttnProcessor,
    SlicedAttnProcessor,
    LoRAAttnProcessor,
    LoRAAttnProcessor2_0,
    LoRAXFormersAttnProcessor,
)

AttentionProcessor = Union[
    AttnProcessor,
    AttnProcessor2_0,
    XFormersAttnProcessor,
    SlicedAttnProcessor,
    AttnAddedKVProcessor,
    SlicedAttnAddedKVProcessor,
    AttnAddedKVProcessor2_0,
    XFormersAttnAddedKVProcessor,
    CustomDiffusionAttnProcessor,
    CustomDiffusionXFormersAttnProcessor,
    # depraceted
    LoRAAttnProcessor,
    LoRAAttnProcessor2_0,
    LoRAXFormersAttnProcessor,
    LoRAAttnAddedKVProcessor,
]

部分出现类似于"AttnProcessor2_0" is not defined的warning

shiyutang commented 9 months ago

@LokeZhou 请问这个PR https://github.com/PaddlePaddle/PaddleMIX/pull/322 是否很快就能合入。

LokeZhou commented 9 months ago

@LokeZhou 请问这个PR #322 是否很快就能合入。

ci通过后合入

LokeZhou commented 9 months ago

diffusers的attention_processor.py里存在的pytorch2.0版本的类需要在ppdiffusers里实现吗 像： AttnAddedKVProcessor2_0，AttnProcessor2_0，LoRAAttnProcessor2_0

如果功能一致不需要；如果是因为2.0版本导致新增功能，则需要。

shiyutang commented 8 months ago

任务已完成 by @co63oc ，close issue。