search

ksugar / samapi

Segment Anything Model API
BSD 3-Clause "New" or "Revised" License
75 stars 12 forks source link

readme

Segment Anything Models (SAM) API

A web API for SAM implemented with FastAPI.

This is a part of the following paper. Please cite it when you use this project. You will also cite the original SAM paper and the MobileSAM paper.

Install

Create a conda environment.

conda create -n samapi -y python=3.10
conda activate samapi

If you're using a computer with CUDA-compatible GPU, install cudatoolkit.

conda install -c conda-forge -y cudatoolkit=11.8

If you're using a computer with CUDA-compatible GPU on Windows, install torch with GPU-support with the following command.

# Windows with CUDA-compatible GPU only
python -m pip install "torch>=2.3.1,<2.4" torchvision --index-url https://download.pytorch.org/whl/cu118

Install samapi and its dependencies.

python -m pip install git+https://github.com/ksugar/samapi.git

If you are using WSL2, LD_LIBRARY_PATH will need to be updated as follows.

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

Update

If you want to update the samapi server, run the following command in the conda environment.

python -m pip install -U git+https://github.com/ksugar/samapi.git

Usage

Launch a server

Since v0.4.0, it is important to launch the server with --workers 2 (or more) to enable cancellation of a download of a weights file.

export PYTORCH_ENABLE_MPS_FALLBACK=1 # Required for running on Apple silicon
uvicorn samapi.main:app --workers 2

The command above will launch a server at http://localhost:8000.

INFO:     Started server process [21258]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

[!NOTE] If you want to access remotely, you may need to launch with --host 0.0.0.0.

uvicorn samapi.main:app --workers 2 --host 0.0.0.0

For more information, see uvicorn documentation.

Troubleshooting

If you try to process a large image and receive the following error, you may need to increase the PIL.Image.MAX_IMAGE_PIXELS value (default: 89478485), or completely disable it (i.e. set the variable to the empty valie).

PIL.Image.DecompressionBombError: Image size (xxxxxxxxx pixels) exceeds limit of 178956970 pixels, could be decompression bomb DOS attack.

In Linux and MacOS, you can set the environment variable as follows.

export PIL_MAX_IMAGE_PIXELS="" # or specific value (integer)

In Windows, you can set the environment variable as follows.

set PIL_MAX_IMAGE_PIXELS="" # or specific value (integer)

Request body

Endpoint /sam/ (post)

class SAMBody(BaseModel):
    type: Optional[ModelType] = ModelType.vit_h
    bbox: Tuple[int, int, int, int] = Field(example=(0, 0, 0, 0))
    b64img: str
key value
type One of vit_h, vit_l, vit_b or vit_t
bbox Coordinate of a bbox (x1, y1, x2, y2)
b64img Base64-encoded image data

Endpoint /sam/automask/ (post)

class SAMAutoMaskBody(BaseModel):
    type: Optional[ModelType] = ModelType.vit_h
    b64img: str
    points_per_side: Optional[int] = 32
    points_per_batch: int = 64
    pred_iou_thresh: float = 0.88
    stability_score_thresh: float = 0.95
    stability_score_offset: float = 1.0
    box_nms_thresh: float = 0.7
    crop_n_layers: int = 0
    crop_nms_thresh: float = 0.7
    crop_overlap_ratio: float = 512 / 1500
    crop_n_points_downscale_factor: int = 1
    min_mask_region_area: int = 0
    output_type: str = "Single Mask"
    include_image_edge: bool = False
key value
type One of vit_h, vit_l, or vit_b.
b64img Base64-encoded image data.
points_per_side The number of points to be sampled along one side of the image. The total number of points is points_per_side**2.
points_per_batch Sets the number of points run simultaneously by the model. Higher numbers may be faster but use more GPU memory.
pred_iou_thresh A filtering threshold in [0,1], using the model's predicted mask quality.
stability_score_thresh A filtering threshold in [0,1], using the stability of the mask under changes to the cutoff used to binarize the model's mask predictions.
stability_score_offset The amount to shift the cutoff when calculated the stability score.
box_nms_thresh The box IoU cutoff used by non-maximal suppression to filter duplicate masks.
crop_n_layers If >0, mask prediction will be run again on crops of the image. Sets the number of layers to run, where each layer has 2**i_layer number of image crops.
crop_nms_thresh The box IoU cutoff used by non-maximal suppression to filter duplicate masks between different crops.
crop_overlap_ratio Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of the image length. Later layers with more crops scale down this overlap.
crop_n_points_downscale_factor The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n.
min_mask_region_area If >0, postprocessing will be applied to remove disconnected regions and holes in masks with area smaller than min_mask_region_area. Requires opencv.
output_type If 'Single Mask' is selected, the model will return single masks per prompt. If 'Multi-mask' is selected, the model will return three masks per prompt. 'Multi-mask (all)' keeps all three masks. One of the three masks is kept if the option 'Multi-mask (largest)', 'Multi-mask (smallest)', or 'Multi-mask (best quality)' is selected.
include_image_edge If True, include a crop area at the edge of the original image.

Response body

The response body contains a list of GeoJSON Feature objects.

Supporting other formats is a future work.

Endpoint /sam/version/ (get)

Returns the version of the SAM API.

Response body

The version of the SAM API.

0.4.1

Endpoint /sam/weights/ (get)

Returns a list of the available weights.

Query parameters
key value
type (Optional) One of vit_h, vit_l, vit_b or vit_t.
Response body

A list of the available weights.

key value
type One of vit_h, vit_l, vit_b or vit_t.
name The name of the registered SAM weights.
URL The URL of the registered SAM weights.

Endpoint /sam/weights/ (post)

Registers SAM weights.

Request body
class SAMWeightsBody(BaseModel):
    type: ModelType
    name: str
    url: str
key value
type One of vit_h, vit_l, vit_b or vit_t.
name The name of the SAM weights to register.
URL The URL to the SAM weights file to register.
Response body

A message indicating whether the registration is successful.

name https://path/to/weights/file.pth is registered.

Endpoint /sam/weights/cancel/ (get)

Cancel the download of the SAM weights.

Response body

A message indicating that the cancel signal is sent.

Cancel signal sent

Endpoint /sam/progress/ (get)

Returns the progress.

Response body

The progress.

key value
message A message indicating the progress.
percent Integer value in [0, 100].

Updates

v0.5.0

v0.4.1

v0.4.0

v0.3.0

v0.2.0

Citation

Please cite my paper on bioRxiv.

@article {Sugawara2023.06.13.544786,
    author = {Ko Sugawara},
    title = {Training deep learning models for cell image segmentation with sparse annotations},
    elocation-id = {2023.06.13.544786},
    year = {2023},
    doi = {10.1101/2023.06.13.544786},
    publisher = {Cold Spring Harbor Laboratory},
    abstract = {Deep learning is becoming more prominent in cell image analysis. However, collecting the annotated data required to train efficient deep-learning models remains a major obstacle. I demonstrate that functional performance can be achieved even with sparsely annotated data. Furthermore, I show that the selection of sparse cell annotations significantly impacts performance. I modified Cellpose and StarDist to enable training with sparsely annotated data and evaluated them in conjunction with ELEPHANT, a cell tracking algorithm that internally uses U-Net based cell segmentation. These results illustrate that sparse annotation is a generally effective strategy in deep learning-based cell image segmentation. Finally, I demonstrate that with the help of the Segment Anything Model (SAM), it is feasible to build an effective deep learning model of cell image segmentation from scratch just in a few minutes.Competing Interest StatementKS is employed part-time by LPIXEL Inc.},
    URL = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786},
    eprint = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.13.544786.full.pdf},
    journal = {bioRxiv}
}

Acknowledgements