search

healthonrails / annolid

An annotation and instance segmentation-based multiple animal tracking and behavior analysis package.
Other
41 stars 9 forks source link

SegmentAnything loading state_dict error #57

Open haolapig opened 1 month ago

haolapig commented 1 month ago

I successfully installed Annolid on Windows by following the documentation:

1) Cloning the GitHub repository 2) Running pip install -e . 3) I also had to reinstall numpy version 1.25.2 because the newer 2.x versions caused size-related issues. 4) I also reinstall pytorch with cuda so that I can utilize my GPU.

The GUI runs well, and I can load videos without any problems. However, when I tried using SAM-based text prompt segmentation by following (https://www.youtube.com/watch?v=ry9bnaajKCs), I encountered an error:

Traceback (most recent call last): File "d:\annolid\annolid\gui\app.py", line 649, in _grounding_sam self.canvas.predictAiRectangle(prompt_text) File "d:\annolid\annolid\gui\widgets\canvas.py", line 274, in predictAiRectangle self.sam_hq_model = SamHQSegmenter() ^^^^^^^^^^^^^^^^ File "d:\annolid\annolid\segmentation\SAM\sam_hq\sam_hq.py", line 41, in init self.sam = sam_model_registrymodel_type ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\miniconda3\envs\annolid-env\Lib\site-packages\segment_anything\build_sam.py", line 28, in build_sam_vit_l return _build_sam( ^^^^^^^^^^^ File "D:\miniconda3\envs\annolid-env\Lib\site-packages\segment_anything\build_sam.py", line 106, in _build_sam sam.load_state_dict(state_dict) File "D:\miniconda3\envs\annolid-env\Lib\site-packages\torch\nn\modules\module.py", line 2215, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Sam: Unexpected key(s) in state_dict: "mask_decoder.hf_token.weight", "mask_decoder.hf_mlp.layers.0.weight", "mask_decoder.hf_mlp.layers.0.bias", "mask_decoder.hf_mlp.layers.1.weight", "mask_decoder.hf_mlp.layers.1.bias", "mask_decoder.hf_mlp.layers.2.weight", "mask_decoder.hf_mlp.layers.2.bias", "mask_decoder.compress_vit_feat.0.weight", "mask_decoder.compress_vit_feat.0.bias", "mask_decoder.compress_vit_feat.1.weight", "mask_decoder.compress_vit_feat.1.bias", "mask_decoder.compress_vit_feat.3.weight", "mask_decoder.compress_vit_feat.3.bias", "mask_decoder.embedding_encoder.0.weight", "mask_decoder.embedding_encoder.0.bias", "mask_decoder.embedding_encoder.1.weight", "mask_decoder.embedding_encoder.1.bias", "mask_decoder.embedding_encoder.3.weight", "mask_decoder.embedding_encoder.3.bias", "mask_decoder.embedding_maskfeature.0.weight", "mask_decoder.embedding_maskfeature.0.bias", "mask_decoder.embedding_maskfeature.1.weight", "mask_decoder.embedding_maskfeature.1.bias", "mask_decoder.embedding_maskfeature.3.weight", "mask_decoder.embedding_maskfeature.3.bias".

healthonrails commented 1 month ago

First, please ensure that your sam-hq model file is located at ./annolid/segmentation/SAM/sam_hq/sam_hq_vit_l.pth. Verify the integrity of the file by checking its MD5 hash:

md5 sam_hq_vit_l.pth

The correct MD5 hash should be:

MD5 (sam_hq_vit_l.pth) = 08947267966e4264fb39523eccc33f86

If the hash matches, the file is downloaded correctly and is not corrupted.

Next, confirm that sam-hq is installed properly in your environment. You can do this by running the following command in your Annolid environment:

pip install git+https://github.com/SysCV/sam-hq.git

This will ensure that sam-hq is installed correctly in your environment.

haolapig commented 1 month ago
haolapig commented 1 month ago

I think I find the solution:

from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

to

from segment_anything_hq import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

Now the text prompt segmentation works. The source of error could be the conflict between original segmenation-anything (facebook git) and segmentation-anyting-hq

healthonrails commented 1 month ago

I think I find the solution:

  • I reinstall sam-hq but instead of pip install git+https://github.com/SysCV/sam-hq.git, I did following: pip install segment-anything-hq I check my conda list and I can see segment-anything-hq now.
  • In /annolid/segmentation/SAM/sam_hq/sam_hq.py, I modified:

from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

to

from segment_anything_hq import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

Now the text prompt segmentation works. The source of error could be the conflict between original segmenation-anything (facebook git) and segmentation-anyting-hq

The setup.py file contains the line "segment-anything @ git+https://github.com/SysCV/sam-hq.git". When you run pip install -e ., it installs the package from the GitHub repository using the alias segment-anything. This line explicitly installs the SAM-HQ package from the SysCV repository but labels it as segment-anything. This prevents any conflict with Meta’s segment-anything package and eliminates the need for a separate package, such as segment-anything-hq.

haolapig commented 1 month ago

The setup.py file contains the line "segment-anything @ git+https://github.com/SysCV/sam-hq.git". When you run pip install -e ., it installs the package from the GitHub repository using the alias segment-anything. This line explicitly installs the SAM-HQ package from the SysCV repository but labels it as segment-anything. This prevents any conflict with Meta’s segment-anything package and eliminates the need for a separate package, such as segment-anything-hq.

As mentioned earlier, I have already tried reinstalling sam-hq using pip install git+https://github.com/SysCV/sam-hq.git, but unfortunately, it did not work for me. The load state_dict continues to reference the original Facebook SAM.

To ensure I hadn't missed anything, I repeated the entire process from scratch on a new Windows machine, and I install Segment Anything specifically from SysCV usingpip install git+https://github.com/SysCV/sam-hq.git. However, I encountered the same error related to load state_dict.

I realized that the issue might have been due to the installation tutorial. As the github tutorial mentioned, before runningpip install -e ., I should have used pip install git+https://github.com/facebookresearch/segment-anything.git. I had skipped this step and pip install -e . directly. Now, everything is working as expected.

In summary, here are the steps I followed to get Annolid working:

  1. Follow the GitHub instructions, but skip the step: pip install git+https://github.com/facebookresearch/segment-anything.git.
  2. Reinstall NumPy version 1.5.2, as NumPy 2.0 causes the following error: ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject.
  3. If you need to use GPU instead of the default CPU, install PyTorch and then reinstall Annolid.
  4. Modify /annolid/segmentation/SAM/sam_hq/sam_hq.py to update the model URL: _REMOTE_MODEL_URL = "https://huggingface.co/lkeab/hq-sam/tree/main/"
healthonrails commented 1 month ago
  1. p install git+https://github.com/facebookresearch/segment-anything.git.

Thank you for sharing your experience! I’m glad to hear that you successfully installed Annolid and it’s working for you. I’m not sure about the environment you’re using to run Annolid, but we recommend using Conda, as outlined in the installation guide: Installation Instructions.