Inferencing with adapter vs Inferencing with fused model

[hbacard]

Hi everyone,

First of all let me say a big "THANK YOU" for your work ! I have successfully fine tuned phi-2. However I noticed the following:

• Inferencing with the fused model usually gives bad results. I use this code python -m mlx_lm.generate --model <fused-model-path> --prompt "hello". I don't see any effect of the fine tuning and sometimes it gives me nonsense.
• However when I use the following code, the results are much much better and I can see the fine tuning: python lora.py --model mlx_model --adapter-file adapters.npz --max-tokens 300 --prompt "<your-prompt-here>"

So I was wondering if someone can help me understand/explain this. My goal is to be able to use the model in a web app.

Thanks a lot !

[awni]

Hmm, some questions:

• Was it a quantized model that you fine-tuned or an fp16 model?
• Is the fused model even worse than the original baseline phi-2?

Sometimes fusing can reduce precision since you are adding a small update to the base model. That's especially the case when you fuse into a quantized model.

But I am not certain that is the issue here. It would be useful to have a little more information to help debug.

[hbacard]

Hi awni,

Thank you for your response :)

• Actually I fine-tuned the model after converting it with the -q flag so I guess it's a quantized model, hence QLoRA.
• I can say that the fused model is worse than the original phi-2.

I am not getting any particular error, it's just that the output isn't exactly what I would expect. What I am going to set up is a set of prompts that I can use to compare the base and the fine-tuned model. I will surely let you know.

[rshah918]

Hey @awni @hbacard I have the same issue. Fused model seems to not have any of my fine tuning in it, but running the model and adaptor seperately works great.

Example:

Quantized model with lora adapter (Good): [INST]how do I paint like bob ross?[/INST]Painting like Bob Ross is refreshing, but let's bring our focus back to REDACTED. What questions do you have about REDACTED?

Fused model (Bad)

 Bob Ross, the famous American painter and television personality, was known for his unique painting style and soothing instructional methods. Here are some steps to help you paint like Bob Ross:

1. Gather your supplies: To paint like Bob Ross, you will need some essential art supplies such as acrylic paints (preferably Liquid Acrylics), brushes (flat, round, fan, and filbert), a palette knife, a canvas or heavy weight watercolor paper, water container, paper towels, and a good quality brush cleaner.

2. Set up your workspace: Find a comfortable and well-lit area to set up your painting station. Make sure all of your supplies are easily accessible.

3. Watch Bob Ross videos for inspiration: There are numerous Bob Ross instructional videos available online. Watching these will give you an idea of his techniques, color choices, and brush strokes.

4. Start with a blank canvas: Begin by applying a thin coat of liquid white paint (called "Magic White" by Bob) to your canvas using a 1-inch flat brush. This creates a smooth, even base for your painting. Allow it to dry completely before proceeding.

5. Plan your composition: Decide on the scene or subject you want to paint. Bob Ross often painted landscapes with trees, water, and animals. Use reference images if needed.

6. Create the sky: Using a clean 1-inch brush, apply titanium white paint to create the base of the sky. Blend in shades of blue, purple, and green using a clean fan brush to create depth and texture.

7. Add trees: Use a 1/2 inch or smaller round brush to add tree trunks. Apply dark brown paint (Phthalo Blue and Alizarin Crimson) in short strokes, then blend in lighter colors for the bark texture. Use a fan brush to create the tree branches.

8. Paint the foliage: Using various shades of green, apply thin layers of paint with a 1/2 inch round brush to create the tree leaves. Bob Ross often used the "happy little trees" technique, where he applied small dots and lines of paint in random patterns to mimic the look of leaves.

9. Add details: Use smaller brushes and fine detail work to add animals, rocks, or other elements to your painting. Blend colors together using clean brushes for a seamless appearance.

10. Finish the painting: Allow your painting to dry completely before applying a final protective coat of varnish or sealant. Enjoy your masterpiece!

This is just one example, but yea basically something goofy is happening during fusing