Do you train only linear classification head or finetune the whole Llama model?

[ngthanhtin]

Hi, Thank you, guys, for providing us with the code, it is very useful for my research.

I have a few questions about your training:

1/ Here, what is the prompt for LS-LLaMA-2-7B? and where can I find this? 2/ Do you train only the linear classification head or fine-tune the whole Llama model? 3/Could you provide us with the zero and few-shot Llama code?

I hope you can answer my question, it is fascinating to find out these.

Best, Tin

[SeanLee97]

hi @ngthanhtin , sorry for the delayed reply.

1) For all LS-* models, we did not use a prompt. 2) We train the linear head and use LoRA for efficient tuning. 3) For the zero or few shot settings, here is our prompt template:

You are an intelligent text classification system; please help me with the classification task.
Task: identify the classification label from the given sentence.
Only support {{ labels|length }} labels, including: {{ labels|join(', ') }}.
{% if hint is not none %}
Explanation and examples: {{ hint }}
{% endif %}
Output format: label.

Following is the given sentence: {{ text }}
Output:

[ngthanhtin]

Thank you so much! @SeanLee97