Closed wj210 closed 10 months ago
If your requirement is to PEFT finetune an LLM on text data without any system prompt, you can make the following data config:
META:
-
path: path/to/your/data.csv
type: 'text'
prompt_type: 'None'
the data.csv
file is expected to contain a column named instruction
, whose contents are your raw text data. Relevant code logics are here:
https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/c7fd8f83d3564e0982c63e8e0a1c8930b30c6cfe/accessory/data/alpaca.py#L115
On the other hand, if your data is large and need the lazy-loading mechanism we support in the pre-training pipeline, you may start from our pre-training examples here, but changing the --llama-type
argument to llama_peft
or llama-adapter
. If you want to start your training from an existing checkpoint, use the --pretrained_path
argument.
Hi, i would like to ask if its possible to do fine-tuning with just raw text (in a pre-training style) with peft? I have a large corpus of text in a specific domain but lacks the ability to transform them into labelled format nor have resource for pretraining.
Would the code support this?