huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.

https://huggingface.co/docs/diffusers

Apache License 2.0

26.35k stars 5.43k forks source link

Update lora_conversion_utils.py #9980

Open zhaowendao30 opened 5 days ago

zhaowendao30 commented 5 days ago

x-flux single-blocks lora load

What does this PR do?

Fixes # (issue)

Before submitting

[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline?
[x] Did you read our philosophy doc (important for complex PRs)?
[ ] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

zhaowendao30 commented 5 days ago

I check the x-flux code, the single block lora should load after qkv attention，not norm,they are just has the same shape

sayakpaul commented 5 days ago

@raulmosa could you give this a look?

raulmosa commented 5 days ago

I've checked it and it's right, it shouldn't be norm. Looks good to me @sayakpaul . Nice catch @zhaowendao30 , thanks! =)

sayakpaul commented 4 days ago

@zhaowendao30 thanks for your contribnutions!

Could you also do a side-by-side comparison of your changes applied and without your changes in the outputs? That would be very much appreciated.

HuggingFaceDocBuilderDev commented 4 days ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

zhaowendao30 commented 4 days ago

I've checked it and it's right, it shouldn't be norm. Looks good to me @sayakpaul . Nice catch @zhaowendao30 , thanks! =)

No thanks =)

zhaowendao30 commented 4 days ago

@zhaowendao30 thanks for your contribnutions!

Could you also do a side-by-side comparison of your changes applied and without your changes in the outputs? That would be very much appreciated.

OK, the single block i just trained index 1,2,3,4, and the first image is load in qkv, the second is load in norm, the last is no lora

sayakpaul commented 4 days ago

@zhaowendao30 thanks, but please prefer not using human subjects in the public forums.

I will run the tests today and update the slices as needed because of the change.

zhaowendao30 commented 4 days ago

@zhaowendao30 thanks, but please prefer not using human subjects in the public forums.

I will run the tests today and update the slices as needed because of the change.

OK, I've uploaded it again. However, this is a LoRA about people, and the panda looks a bit strange. =）

sayakpaul commented 4 days ago

Just ran

pytest tests/lora/ -k "test_flux_xlabs"

test_flux_xlabs is passing and test_flux_xlabs_load_lora_with_single_blocks is failing because of hardware change, which is expected. I will change the slices in https://github.com/huggingface/diffusers/pull/9845/files.