search

facebookresearch / segment-anything

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Apache License 2.0
47.69k stars 5.64k forks source link

nonzero MAX_INT #641

Open schamori opened 11 months ago

schamori commented 11 months ago

Error Summary:

Type of Error: RuntimeError in PyTorch.

Function Causing Error: nonzero function.

Description: The error occurs when the nonzero function is applied to a tensor with more than INT_MAX elements, where INT_MAX represents the maximum value of a 32-bit integer (2,147,483,647). This is probabl

Context: The error is encountered during the run-length encoding (RLE) process of mask data in the mask_to_rle_pytorch function, which is part of an image processing pipeline from SAM.

Potential Causes:

Given that the input images consist of 180,652,032 elements, they fall within the 32-bit size range. However, SAM is using mask_to_rle_pytorch function with multiple masks as input, leading to the resulting size exceeding the 32-bit limit, if there are more than 10 masks.

Potential Fixes:

Without modifying the segment_anything code, the only solution is to reduce the quality of our images. Here are the sizes of the images before and after applying a zoom of 0.4:

Image Size in byte before zoom of 0.4: 180652032 Image Size in byte after zoom of 0.4: 28902804 Please note that a zoom factor of 0.4 is the minimum requirement for the program to run. Differnt hyperparamters seem to have no affect on the minimum zoom required, probably due to a fixed mask batch size, that is passed through mask_to_rle_pytorch.

Since we are now using SamAutomaticMaskGenerator a lot is done in the background, which we cannot control. After training our own model, we could probably influence the batch size of the masks.

Notes After encountering this bug, GPU memory is not freed. Running it a second time results in another error:

OutOfMemoryError: CUDA out of memory. Tried to allocate 43.07 GiB. GPU 0 has a total capacty of 79.15 GiB of which 1.55 GiB is free. Process 2327113 has 77.60 GiB memory in use. Of the allocated memory 26.90 GiB is allocated by PyTorch, and 50.20 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

heyoeyo commented 11 months ago

If you don't mind modifying the code, there is a related issue (#554) and corresponding code fix/pull request (#569) that may fix the problem. If you don't want to modify the code, you may consider trying to run everything on CPU, since the error is apparently GPU-specific, though this will obviously run a lot slower!