search

facebookresearch / sam2

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Apache License 2.0
12.14k stars 1.1k forks source link

sam2 real time inference? #388

Open Karmik25 opened 3 weeks ago

Karmik25 commented 3 weeks ago

can someone explain to me how to use sam2 for real-time inference, such that the model can predict objects while my webcam is open.

heyoeyo commented 2 weeks ago

The existing code doesn't support live/streaming video. Instead you'd have to first record a video on webcam and then process that video using the existing video predictor and/or web demo. It is possible to handle streaming video by modifying the code. It's mostly a matter of re-implementing the existing frame loop outside of the sam2_video_predictor.py script, and handling the frame reading (e.g. from a webcam) in that new loop.

If you don't want to implement this manually, there is an existing repo that has done this: https://github.com/Gy920/segment-anything-2-real-time I also have a script that works with webcams (though it's worth noting the repo above has code much more similar to the original, in case that's important): https://github.com/heyoeyo/muggled_sam?tab=readme-ov-file#run-video-or-webcam