SAM 2.1 Model Loading Error (Unexpected key(s) in state_dict)

[HarryWangATX]

I currently am trying to use the SAM2.1 models. I downloaded the sam2.1_hiera_large.pt and pasted in the sam2.1_hiera_l.yaml config file. However, when I am getting the predictor with predictor = SAM2ImagePredictor(build_sam2(model_cfg, checkpoint)). I am getting the following error:

RuntimeError: Error(s) in loading state_dict for SAM2Base:
        Unexpected key(s) in state_dict: "no_obj_embed_spatial", "obj_ptr_tpos_proj.weight", "obj_ptr_tpos_proj.bias". 

Has anyone encountered/know how to solve this error? Thanks!

[ronghanghu]

Hi @HarryWangATX, have you also pulled the latest code from the repo? The SAM 2.1 checkpoints also require using the latest code in this repository, which includes a few updated module components.

[wooferclaw]

Same issue for me - using code from latest main, trying to use sam2.1_hiera_small with corresponding yaml,

Commit ref from my Pipfile.lock

        "sam-2": {
            "git": "https://github.com/facebookresearch/segment-anything-2.git",
            "ref": "429a2c736032d38af679049a0c76bd962e51cb3c"
        },

[ronghanghu]

Hi @wooferclaw @HarryWangATX

While we cannot reproduce this issue on our end, I suspect the old code is still used (due to a previous installation) even after pulling the latest code. Could you try the following?

1) after pulling the latest code from this repo, run pip uninstall -y SAM-2; pip uninstall -y SAM-2 (same command twice) to uninstall any previous installations, and then install again using pip install -e ".[notebooks]"

2) in case (1) above still doesn't resolve the issue, could you run in your python environment the following

from sam2.modeling import sam2_base

print(sam2_base.__file__)

and check if the content in the printed local file path matches the latest one in https://github.com/facebookresearch/segment-anything-2/blob/main/sam2/modeling/sam2_base.py (e.g. does your local file have no_obj_embed_spatial)?

[HarryWangATX]

Hi @ronghanghu,

Thanks for the help! I have pulled the latest change from the repo and reinstalled my package. However, my code is still running into the same problem. I have diff checked my sam2_base.py (it contains the no_obj_embed_spatial too) file with the one hosted on this repository and there were no differences.

The commit I pulled was 05d9e57fb3945b10c861046c1e6749e2bfc258e3

[ronghanghu]

@HarryWangATX This is weird and I don't have an immediate explanation. To help figure out the issue, could you check the following?

• Could you confirm that you're using the new config file for sam2.1_hiera_l.yaml in build_sam2 or build_sam2_video_predictor, which contains the line no_obj_embed_spatial: true in YAML config? (It should have this line if the code is correctly setup)
• In your local sam2_base.py file (the path from from sam2.modeling import sam2_base; print(sam2_base.__file__)), could you try adding a line assert no_obj_embed_spatial before this line https://github.com/facebookresearch/sam2/blob/05d9e57fb3945b10c861046c1e6749e2bfc258e3/sam2/modeling/sam2_base.py#L177 and see if the assertion fails in your model initialization? (The assertion should not fail if the code is correctly setup)
• Also in your local sam2_base.py file (the path from from sam2.modeling import sam2_base; print(sam2_base.__file__)), could you try adding a line print("model parameters:", [n for n, _ in self.named_parameters()]) before this line https://github.com/facebookresearch/sam2/blob/05d9e57fb3945b10c861046c1e6749e2bfc258e3/sam2/modeling/sam2_base.py#L184 and see if the printed list of model parameters contains no_obj_embed_spatial? (The printed list should contain a parameter named no_obj_embed_spatial if the code is correctly setup)

[HarryWangATX]

Hi @ronghanghu , thank you for the help! I have resolved the issue. Turns out after I installed, it did not update the config files for me, so I was still using the old config files. I copied in the new ones (sam2.1_hiera_l.yaml), and it runs now.