generate/lora is tied to Alpaca instruction style

[DavidGOrtega]

The script is tied to Alpaca prompt style. I can understand that basically all the prepare scripts enforces Alpaca style, however generate/base is agnostic (for good!)

prompt = generate_prompt(sample)

[carmocca]

This is because the finetune/lora.py is tied to the alpaca instruction style too, and that is the generation script linked to finetune/lora.py.

What would you suggest we do instead?

[DavidGOrtega]

This is because the finetune/lora.py is tied to the alpaca instruction style too,

Is it? aside this line that could be whatever, whats the other dependency? Its going to swallow any train.pt and eval.pt

What would you suggest we do instead?

Leave it as base, I mean just pass the instruction style prompt

python generate/base.py  --checkpoint_dir $OUT_DIR/ --prompt "### Instruction:\nList three rivers in Europe\n\n### Response:"

python generate/lora.py  --checkpoint_dir $OUT_DIR/ --prompt "### Instruction:\nList three rivers in Europe\n\n### Response:"

[carmocca]

That's an interesting solution, however, there's one more necessary change which is importing the LoRA version of the model: https://github.com/Lightning-AI/lit-gpt/blob/74b8df0c3f07fc31d9d1a49e870a1f7955329ad8/generate/lora.py#L16 and merging the weights if necessary: https://github.com/Lightning-AI/lit-gpt/blob/74b8df0c3f07fc31d9d1a49e870a1f7955329ad8/generate/lora.py#L125

[DavidGOrtega]

sorry @carmocca I do not follow it. Why is that related to Alpaca? The only change needed would be just remove the Alpaca prompter, right?

[carmocca]

If you lora-finetuned a model without following the alpaca prompt format, and you want to try generation, then yes, you would only need to remove the alpaca format from the generation script. You would still want to use the generate/lora.py script to use the correct model definition.

[DavidGOrtega]

Thats the purpose of this issue 🙂 . To be able to generate with non Alpaca lora weights not having to merge them.

[awaelchli]

We could go down the HF path and save the instruction template as part of the checkpoint/config file (generation_config). Then any script that loads a checkpoint would have to read and apply the template. This would make it portable but of course it introduces more boilerplate logic everywhere. The direction lit-gpt is going, this is probably going to be necessary sooner or later though.

[carmocca]

I'm not sure it should be config-based as that will be limited in supporting multi-turn conversations (related #308)

As a v1, I would suggest adapting https://github.com/Lightning-AI/lit-gpt/blob/f20c431605427c70b5a44a760a923a8adb37cc1d/chat/base.py#L191-L365 and building on top so that prompt configs and conversation configs can be programmatically written and used in different scripts. With the requirement that they are supported directly the function(s) instead of making them externally configurable

[DavidGOrtega]

Sorry for the late reply. I had an accident prior to Christmas and I did not have the chance to contribute to this repo.

if re.search(r"TinyLlama.*Chat", checkpoint_name):

I would stay totally away of this just because the instruction prompt is not tied to the model itself, depends on the dataset i.e. you can find mistral with chatml, zephyr, etc...

To me what should be done (and I can do) is leave the prompt empty and leave the user to feed the desired prompt.