More GPU memory for new mistrail version..

[yuanrr]

When I finetuned videochat2-mistral on 4090, I found that the mistral version was always out of memory. It was probably due to the use of lora for more layers (originally just q and v).

Could you please give me some suggestions about how to finetune the new version in about 24g GPU memory....thanks a lot : )

[Andy1621]

Can you try to freeze the lora and LLM? If it's okay, I think you can load the original LoRA, and only update part of them.

[yuanrr]

I tried the following changes and it worked, thanks~ Plus, would you consider modifying the 8-bit training of videochat model? or do you have any suggestions? Thank you very much!

        if self.use_lora:
            logger.info("Use lora")
            peft_config = LoraConfig(
                task_type=TaskType.CAUSAL_LM, inference_mode=False, 
                r=lora_r, lora_alpha=lora_alpha, lora_dropout=lora_dropout,
                target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                 "gate_proj", "up_proj", "down_proj", "lm_head"]
            )
            self.mistral_model = get_peft_model(self.mistral_model, peft_config)
            # self.mistral_model.print_trainable_parameters()

            # TODO
            unfreeze_patterns = {'self_attn.v_proj.lora', 'self_attn.q_proj.lora'}
            for name, param in self.mistral_model.named_parameters():
                if not any(pattern in name for pattern in unfreeze_patterns):
                    param.requires_grad = False

[Andy1621]

Thanks for your good try! I have not tried to adopt 8-bit training seriously, since it leads to unstable training when I tried a few months ago and then stop the exploration.

[yuanrr]

Thank you for your kind reply : )