inference error

[setayeshk]

Hello,

I encountered some errors while trying to run inference with the following command:

!python bctv_test.py -net sam2 -exp_name BTCV_MedSAM2 -sam_ckpt {base_path}/Medical-SAM2/sam2_hiera_small.pt -sam_config sam2_hiera_s.yaml -image_size 1024 -val_freq 1 -prompt bbox -prompt_freq 2 -dataset btcv -data_path ./data/btcv

The specific errors I encountered are as follows: /kaggle/working/Medical-SAM2/sam2_train/modeling/sam/transformer.py:22: UserWarning: Flash Attention is disabled as it requires a GPU with Ampere (8.0) CUDA capability. OLD_GPU, USE_FLASH_ATTN, MATH_KERNEL_ON = get_sdpa_settings() Validation round: 0%| | 0/6 [00:00<?, ?batch/s] /opt/conda/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning: os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock. self.pid = os.fork() Traceback (most recent call last):
File "/kaggle/working/Medical-SAM2/func_3d/function.py", line 263, in validation_sam bbox = bbox_dict[id][ann_obj_id] KeyError: 2.0

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/kaggle/working/Medical-SAM2/bctv_test.py", line 52, in main(args) File "/kaggle/working/Medical-SAM2/bctv_test.py", line 38, in main validation_loss, validation_metrics = validation_sam(args, nice_test_loader, 0, net) File "/kaggle/working/Medical-SAM2/func_3d/function.py", line 273, in validationsam , , = net.train_add_new_mask( File "/kaggle/working/Medical-SAM2/sam2_train/sam2_video_predictor.py", line 692, in train_add_new_mask currentout, = self._run_single_frame_inference( File "/kaggle/working/Medical-SAM2/sam2_train/sam2_video_predictor.py", line 1351, in _run_single_frame_inference pred_masks_gpu = fill_holes_in_mask_scores( File "/kaggle/working/Medical-SAM2/sam2_train/utils/misc.py", line 254, in fill_holes_in_mask_scores labels, areas = get_connected_components(mask <= 0) File "/kaggle/working/Medical-SAM2/sam2_train/utils/misc.py", line 61, in get_connected_components from sam2_train import _C ImportError: /kaggle/working/Medical-SAM2/sam2_train/_C.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKSs

I noticed that similar _C.so issues have been mentioned in the SAM2 repository. As I understand it, one possible solution involves recompiling the C++ extensions using a setup.py file to ensure compatibility with the current version of PyTorch. However, I couldn't find a setup.py file in your project.

could you please help me with this

[saugatkandel]

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory.

Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there.

I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

[jiayuanz3]

thanks for providing the solution and arising the issue here. we'll try the best to improve the code base and apologise for any inconvenience brought to you.

[setayeshk]

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory.

Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there.

I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

Thank you so much for the suggestion.

I’m running in kaggle and i don’t know what to do to fix this problem there. I tried cloning the sam2 project and installing their requirements but I’m getting a conflict when I run this line: !pip install -e ".[demo]"

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. beatrix-jupyterlab 2024.66.154055 requires jupyterlab~=3.6.0, but you have jupyterlab 4.2.4 which is incompatible. pointpats 2.5.0 requires shapely>=2, but you have shapely 1.8.5.post1 which is incompatible. spopt 0.6.1 requires shapely>=2.0.1, but you have shapely 1.8.5.post1 which is incompatible. ydata-profiling 4.9.0 requires scipy<1.14,>=1.4.1, but you have scipy 1.14.0 which is incompatible.

Do you know how I could fix this issue on Kaggle? Thanks again for your help

[saugatkandel]

I haven't worked with Kaggle, so it might be better if you ask this in the SAM2 repository. The audience base there is very large.

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory. Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there. I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

Thank you so much for the suggestion.

I’m running in kaggle and i don’t know what to do to fix this problem there. I tried cloning the sam2 project and installing their requirements but I’m getting a conflict when I run this line: !pip install -e ".[demo]"

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. beatrix-jupyterlab 2024.66.154055 requires jupyterlab~=3.6.0, but you have jupyterlab 4.2.4 which is incompatible. pointpats 2.5.0 requires shapely>=2, but you have shapely 1.8.5.post1 which is incompatible. spopt 0.6.1 requires shapely>=2.0.1, but you have shapely 1.8.5.post1 which is incompatible. ydata-profiling 4.9.0 requires scipy<1.14,>=1.4.1, but you have scipy 1.14.0 which is incompatible.

Do you know how I could fix this issue on Kaggle? Thanks again for your help

[WangZhen-Ryan]

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory.

Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there.

I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

It worked for me!!!

[setayeshk]

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory. Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there. I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

It worked for me!!!

Did you run this on Kaggle or on your local machine?

[saugatkandel]

Just in case this helps anyone reading this thread:

The root of the issue is a mismatch between the local version of CUDA and the CUDA version used to compile PyTorch for SAM2.

If you install the SAM2 repository from scratch, without using a previously installed PyTorch, then it uses Pytorch compiled with CUDA 12.1 (I think). So if your local CUDA version is different, then the _C.so file is not created in the setup step, so you have to work around it somehow.

I think the options are to either:

• use CUDA 12.1 locally, or
• first install PyTorch compiled with a matching CUDA version, then install SAM2. For example, there are PyTorch with CUDA 12.4 is available either in PyTorch conda channel, or as a wheel in pip.

With CUDA 12.1, the _C.so file that comes with Medical-SAM2 might also work out of the box, with no extra steps.

[WangZhen-Ryan]

I addressed the _C issue by installed the SAM2 repository separately by:

1. Creating a new conda environment and installing https://anaconda.org/nvidia/cuda and python.
2. Installing SAM2 in this environment.

Then, I replaced the _C file in the Meidcal-SAM2 directory with a hard link to the same file in the SAM2 directory. Generally, I feel like the Medical-SAM2 repository is a bit bloated. Most of it is reproducing the SAM2 repo with small changes here and there. I would be very grateful if the developers could build this repo using SAM2 as an import, rather than as a variation of the original SAM2 code. That would make incorporating any upstream SAM2 changes much easier.

Thank you so much for the suggestion.

I’m running in kaggle and i don’t know what to do to fix this problem there. I tried cloning the sam2 project and installing their requirements but I’m getting a conflict when I run this line: !pip install -e ".[demo]"

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. beatrix-jupyterlab 2024.66.154055 requires jupyterlab~=3.6.0, but you have jupyterlab 4.2.4 which is incompatible. pointpats 2.5.0 requires shapely>=2, but you have shapely 1.8.5.post1 which is incompatible. spopt 0.6.1 requires shapely>=2.0.1, but you have shapely 1.8.5.post1 which is incompatible. ydata-profiling 4.9.0 requires scipy<1.14,>=1.4.1, but you have scipy 1.14.0 which is incompatible.

Do you know how I could fix this issue on Kaggle? Thanks again for your help

@setayeshk It is on local machine.

[setayeshk]

Hi,

Thanks everybody for helping me.. I finally got everything working and wanted to share the steps I followed:

I cloned both the SAM2 and Medical SAM2 repositories. Instead of installing the exact versions of the requirements as mentioned in the setup.py, I installed the following packages:

!pip install torch torchvision numpy tqdm hydra-core iopath pillow # Required packages !pip install matplotlib jupyter opencv-python # Extra packages

I created the _C.so file based on the PyTorch and CUDA versions available of Kaggle by : !python setup.py build_ext --inplace

Finally, I replaced the _C.so file from the Medical SAM2 repository with the one from the SAM2 repository.

Just in case this helps anyone reading this thread:

The root of the issue is a mismatch between the local version of CUDA and the CUDA version used to compile PyTorch for SAM2.

If you install the SAM2 repository from scratch, without using a previously installed PyTorch, then it uses Pytorch compiled with CUDA 12.1 (I think). So if your local CUDA version is different, then the _C.so file is not created in the setup step, so you have to work around it somehow.

I think the options are to either:

• use CUDA 12.1 locally, or
• first install PyTorch compiled with a matching CUDA version, then install SAM2. For example, there are PyTorch with CUDA 12.4 is available either in PyTorch conda channel, or as a wheel in pip.

With CUDA 12.1, the _C.so file that comes with Medical-SAM2 might also work out of the box, with no extra steps.

[godspeedlucip]

Hi,

Thanks everybody for helping me.. I finally got everything working and wanted to share the steps I followed:

I cloned both the SAM2 and Medical SAM2 repositories. Instead of installing the exact versions of the requirements as mentioned in the setup.py, I installed the following packages:

!pip install torch torchvision numpy tqdm hydra-core iopath pillow # Required packages !pip install matplotlib jupyter opencv-python # Extra packages

I created the _C.so file based on the PyTorch and CUDA versions available of Kaggle by : !python setup.py build_ext --inplace

Finally, I replaced the _C.so file from the Medical SAM2 repository with the one from the SAM2 repository.

Just in case this helps anyone reading this thread: The root of the issue is a mismatch between the local version of CUDA and the CUDA version used to compile PyTorch for SAM2. If you install the SAM2 repository from scratch, without using a previously installed PyTorch, then it uses Pytorch compiled with CUDA 12.1 (I think). So if your local CUDA version is different, then the _C.so file is not created in the setup step, so you have to work around it somehow. I think the options are to either:

• use CUDA 12.1 locally, or
• first install PyTorch compiled with a matching CUDA version, then install SAM2. For example, there are PyTorch with CUDA 12.4 is available either in PyTorch conda channel, or as a wheel in pip.

With CUDA 12.1, the _C.so file that comes with Medical-SAM2 might also work out of the box, with no extra steps.

Thanks a lot! My code runs smoothly following your step!

[ArjunBalaji79]

Hi,

Thanks everybody for helping me.. I finally got everything working and wanted to share the steps I followed:

I cloned both the SAM2 and Medical SAM2 repositories. Instead of installing the exact versions of the requirements as mentioned in the setup.py, I installed the following packages:

!pip install torch torchvision numpy tqdm hydra-core iopath pillow # Required packages !pip install matplotlib jupyter opencv-python # Extra packages

I created the _C.so file based on the PyTorch and CUDA versions available of Kaggle by : !python setup.py build_ext --inplace

Finally, I replaced the _C.so file from the Medical SAM2 repository with the one from the SAM2 repository.

Just in case this helps anyone reading this thread: The root of the issue is a mismatch between the local version of CUDA and the CUDA version used to compile PyTorch for SAM2. If you install the SAM2 repository from scratch, without using a previously installed PyTorch, then it uses Pytorch compiled with CUDA 12.1 (I think). So if your local CUDA version is different, then the _C.so file is not created in the setup step, so you have to work around it somehow. I think the options are to either:

• use CUDA 12.1 locally, or
• first install PyTorch compiled with a matching CUDA version, then install SAM2. For example, there are PyTorch with CUDA 12.4 is available either in PyTorch conda channel, or as a wheel in pip.

With CUDA 12.1, the _C.so file that comes with Medical-SAM2 might also work out of the box, with no extra steps.

This worked for me as well, Thank you !