Any method to finetune embedding layers using Xtuner?

[fan2goa1]

I want to add new tokens to expand the vocab and resize the embedding of a LLM model. And I'm wonder if I can use Xtuner to finetune the embedding layer of the LLM?

[fan2goa1]

I've figured out that I can edit the configuration file, such as internlm2_chat_20b_qlora_alpaca_copy.py.

model = dict(
    type=SupervisedFinetune,
    use_varlen_attn=use_varlen_attn,
    llm=dict(
        type=AutoModelForCausalLM.from_pretrained,
        pretrained_model_name_or_path=pretrained_model_name_or_path,
        trust_remote_code=True,
        torch_dtype=torch.float16,
        quantization_config=dict(
            type=BitsAndBytesConfig,
            load_in_4bit=True,
            load_in_8bit=False,
            llm_int8_threshold=6.0,
            llm_int8_has_fp16_weight=False,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type='nf4')),
    lora=dict(
        type=LoraConfig,
        target_modules=['wqkv', 'w1', 'w3', 'wo', 'w2', 'output', 'tok_embeddings'],      # zzf add: train embed
        r=64,
        lora_alpha=16,
        lora_dropout=0.1,
        bias='none',
        task_type='CAUSAL_LM'))

We can add the layers that we want to finetune to the parameter target_modules.