aredden / flux-fp8-api

Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.
Apache License 2.0
209 stars 22 forks source link

LoRA loaded successfully but the effect wasn't applied #28

Open EntroSanity opened 1 month ago

EntroSanity commented 1 month ago

I was testing the pipeline's capability to apply a Flux LoRA effect, but I couldn't achieve the desired effect in my generated images. I experimented with two different methods. Both methods successfully displayed a "LoRA successfully loaded" message; however, the generated images did not reflect the applied LoRA effect. Could I get some assistance in case I'm missing any crucial steps?

  1. Using /lora endpoint:
    
    import requests
    import json

Load the LoRA

lora_data = { "path": "path-to-my-lora", "action": "load", "scale": 1.0, # Adjust this value as needed, "name": None # Include this even if it's None }

response = requests.post("http://localhost:8088/lora", json=lora_data) print(response.json())


2. Using flux pipeline:
```python
from flux_pipeline import FluxPipeline

config_path = "path/to/config/file.json"
config_overrides = {
    #...
}
lora_path = "path/to/lora/file.safetensors"
pipeline = FluxPipeline.load_pipeline_from_config_path(config_path, **config_overrides)
pipeline.load_lora(lora_path, scale=1.0)
## 
run inference
aredden commented 1 month ago

I have actually been thinking that I've implemented the qkv loras a little incorrectly, which I do plan to fix, though requires a bit of an overhaul since I have to keep track of exactly which section of the fused qkv weight to apply unfused lora weights to. Though also, are you using the current version? There was a pretty significant improvement in lora implementation that I did recently (aka like two weeks ago or so).

EntroSanity commented 3 weeks ago

I initially believed that the LoRA scale was limited to a range of 0 to 1, but I noticed that within this range, it didn't seem to affect the generated image. However, when I tried using a scale larger than 1, it successfully applied the LoRA effect to the image.

lora_data = {
    "path": "path-to-my-lora",
    "action": "load",
    "scale": 1.5,  # Adjust this value as needed,
    "name": None  # Include this even if it's None
}