search

zhuyiche / llava-phi

361 stars 38 forks source link

What LORA settings are optimal for smaller models? #21

Open awaisrauf opened 3 months ago

awaisrauf commented 3 months ago

I was going through the code and saw that the default LORA parameters are different from the LLaVA implementation. Is there any specific reason for it?

Link to the line: link.

Code snippet. 

    lora_enable: bool = False
    lora_r: int = 64
    lora_alpha: int = 16
    lora_dropout: float = 0.05
    lora_weight_path: str = ""
    lora_bias: str = "none"
    non_lora_lr: Optional[float] = None
    group_by_modality_length: bool = field(default=False)