search

livepeer / ai-worker

https://livepeer.ai
MIT License
17 stars 27 forks source link

Segment anything 2 pipeline image #165

Closed pschroedl closed 3 weeks ago

pschroedl commented 1 month ago

This pull request introduces an implementation of the Segment Anything v2 (SAM2) pipeline within the AI worker.

This implements the basic functionality needed to perform segmentation on an image, including specifying points and labels. Returns the scores, logits, and low-res-masks.

In addition, this PR introduces a new method of adding pipelines dependencies without breaking existing implementations present in ai-runner. Example usage for local development:

cd ai-worker/runner
docker build -t livepeer/ai-runner:base .
docker build -f ../cmd/segment-anything-2/Dockerfile.segment_anything_2 -t livepeer/ai-runner:segment-anything-2 .

docker run --name sam2-runnner -e MODEL_DIR=/models -e PIPELINE=segment-anything-2 -e MODEL_ID=facebook/sam2-hiera-large -e HUGGINGFACE_TOKEN={token} --gpus 0 -p 8002:8000 -v ~/.lpData/models:/models livepeer/ai-runner:segment-anything-2

The existing Dockerfile is specified as a 'base' image ( e.g. : FROM livepeer/ai-runner:base ) in cmd/segment-anything-2/Dockerfile.segment_anything_2.

The go-livepeer PR here implements the logic necessary to launch this pipeline-specific container when appropriate : https://github.com/livepeer/go-livepeer/pull/3131