Question: Where is the setting to freeze the backbone LLM like LLaVA?

YuanGongND / ltu

Code, Dataset, and Pretrained Models for Audio and Speech Large Language Model "Listen, Think, and Understand".

389 stars 36 forks source link

# LLaVA
if model_args.freeze_backbone:
        model.model.requires_grad_(False)

In LTU code, only note the LLM has already frozen.

    # for audio params, lora always trainable, llama always frozen
    for name, param in model.named_parameters():
        if trainable_params == 'all':
            if "audio" in name:
                param.requires_grad = True
                #print(f"Parameter: {name}, requires_grad: {param.requires_grad}")
        if trainable_params == 'proj':
            if "audio_proj" in name:
                param.requires_grad = True
                #print(f"Parameter: {name}, requires_grad: {param.requires_grad}")

YuanGongND / ltu

Question: Where is the setting to freeze the backbone LLM like LLaVA? #47