Is it same the "VQA_lora_PMC_LLaMA_PMCCLIP" and "QA_PMC_LLaMA_lora_PMC-CLIP_MLP"?

[Junst]

I want to run test.py in MedVInT_TD directory. But when I changed the path of ckp(VQA_lora_PMC_LLaMA_PMCCLIP/choice/checkpoint-4000) and model path (chaoyi-wu/PMC_LLAMA_7B), and tokenizer_path to chaoyi-wu/PMC_LLAMA_7B, it doesn't work

I can't find the dir of QA_PMC_LLaMA_lora_PMC-CLIP_MLP.

[xiaoman-zhang]

Line 9 in test.py should change to ``from model.QA_model import QA_model''. The released model checkpoint used transformer-based trainable projection module.

[Junst]

Thank you for your commend, also you have to change 'from model.QA_model -> from models.QA_model' if that is right. If I have a any problem, I will add the issue on this. Thanks.

[xiaoman-zhang]

Thanks for your kind reminding. I have updated the code.

[Junst]

I still have a problem same with first issue. I use "VQA_lora_PMC_LLaMA_PMCCLIP/choice/checkpoint-4000/pytorch_model.bin" in test.py 21 line for ckp import. The model_path is from "llama-7b-hf", visual_model_path is from pmc_clip's checkpoint.pt.

The problem comes from test.py line 110 "model.load_state_dict(torch.load(ckp, map_location='cpu'))". For example, every key from "VQA_lora_PMC_LLaMA_PMCCLIP/choice/checkpoint-4000/pytorch_model.bin" is looks like "llamacasual.base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight" , but expected key is "llamacasual.base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight" which has not "default" in key. How can I solve it?

[nhandang-ai4ia]

Hi, also got the same problem, when i download the pytorch_model.bin and tried test.py MedVInT_TD directory. Manage to fix it by directly rename and delete some keys. Here is my hacky way of fixing it:

ckp = model_args.ckp + "/pytorch_model.bin"
modelCheckpoint = torch.load(ckp, map_location="cpu")
for key in list(modelCheckpoint.keys()):
    if "llamacasual.base_model.model.model.layers":
        if "lora_A.weight" in key:
            # key = key.replace("lora_A.weight", "lora_A.default.weight")
            modelCheckpoint[key.replace("lora_A.weight", "lora_A.default.weight")] = modelCheckpoint.pop(key)
        elif "lora_B.weight" in key:
            modelCheckpoint[key.replace("lora_B.weight", "lora_B.default.weight")] = modelCheckpoint.pop(key)
        elif "rotary_emb.inv_freq" in key:
            del modelCheckpoint[key]

Please check if it also fix on your side, and if you have time please check if it delete any extra keys. I will need to check if this can actually run through to the end or not.

[zhongzee]

@xiaoman-zhang have you deal with this problem?