Implement ELLA for SD1.5

finegrain-ai / refiners

A microframework on top of PyTorch with first-class citizen APIs for foundation model adaptation

https://refine.rs

MIT License

585 stars 43 forks source link

Implement ELLA for SD1.5 #389

Closed ily-R closed 1 week ago

ily-R commented 1 month ago

This is my draft PR for the bounty ELLA for SD1.5. I implemented and tested the architecture. I have few steps left.

[x] Implement ELLA
[x] Test the adapter's inject/eject
[x] implement the weight conversion script for these weights
[x] test ELLA contribution to SD1.5 model using t5_xl text_embeddings from here

github-actions[bot] commented 1 month ago

This bounty is stale because it has been opened for 7 days with no activity.

Laurent2916 commented 1 month ago

I've run the tests, beside the fact that sentencepiece was missing from the dependencies, the 3 tests are green.

github-actions[bot] commented 3 weeks ago

This bounty is stale because it has been opened for 7 days with no activity.

deltheil commented 3 weeks ago

I have nothing to add (except linting and rebase);

That's great!

@deltheil, could you have a final look?

OK, stay tuned.

Also, what about the sentencepiece dependency?

@limiteinductive where is it used? ("[...] beside the fact that sentencepiece was missing from the dependencies, the 3 tests are green")

Laurent2916 commented 3 weeks ago

Hi @ily-R, final nits before we merge:

could you please add sentencepiece as a test dependency
because of a recent PR, there a small conflict in src/refiners/foundationals/latent_diffusion/stable_diffusion_1/__init__.py, could please resolve it

ily-R commented 2 weeks ago

I rebased and did the necessary lint/format and added the sentencepiece as a test dependency . The tests are green. Let me know if there is something to fix

deltheil commented 1 week ago

@ily-R thanks for the quick turnaround. Some last suggestions (see above).