search

Amelie-Schreiber / esm2_loras

Trying to train LoRAs for ESM-2
https://huggingface.co/AmelieSchreiber/esm2_t6_8M_UR50D_lora_rna_binding_sites
7 stars 0 forks source link

Incorrectly loading trained weights #1

Open jamaliki opened 1 year ago

jamaliki commented 1 year ago

Hi, this is Kiarash.

So you are incorrectly loading your already trained weights. LoRA weights are not the same as other HF weights, you need to load them on top of your base model. This was already described in the link I sent you, but for your convenience, I am sharing the exact lines of code you would need to run.

from transformers import AutoModelForTokenClassification, AutoTokenizer
from peft import PeftModel

# Path to the saved LoRA model
model_path = "esm2_t6_8M-finetuned-lora_2023-08-03_18-32-25"
# ESM2 base model
base_model_path = "facebook/esm2_t6_8M_UR50D"

# Load the model
base_model = AutoModelForTokenClassification.from_pretrained(base_model_path)
loaded_model = PeftModel.from_pretrained(base_model, model_path)

# Load the tokenizer
loaded_tokenizer = AutoTokenizer.from_pretrained(model_path)
Amelie-Schreiber commented 1 year ago

Thank you! I think there is still an issue either during training or with the way I'm running the model. It seems like it thinks everything is a non-binding site for some reason. I really can't say why considering it seems to get about 95% accuracy. I thought this might be due to an imbalance in the number of non-binding sites vs. binding sites. However, I trained a new model to take this into account and I am still having the same issue. (see this new training script)