search

facebookresearch / sam2

The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Apache License 2.0
12.43k stars 1.15k forks source link

Making SAM 2 run 2x faster #271

Open mvoodarla opened 2 months ago

mvoodarla commented 2 months ago

I was pretty amazed with SAM 2 when it came out given all the work I do with video. My company works a ton with it and we decided to take a crack at optimizing it, and we made it run 2x faster than the original pipeline!

Unlike LLMs, video models are notorious for incredibly inefficient file reading, storage, and writing which makes them much slower than they need to be.

We wrote a bit about our work here and thought we'd share with the community: https://www.sievedata.com/blog/meta-segment-anything-2-sam2-introduction

Zarxrax commented 2 months ago

Do you have any details on exactly what was done to make it perform faster?

mvoodarla commented 2 months ago

A lot of it is related to how files are read, stored, and saved by the pipeline around the model though there were also some compute / system-level performance gains we made. Hopefully we will do a separate blog post soon that focuses more on these technical details. Generally though, we've found that most video models tend to be really inefficient in how they're reading frames and performing operations over them in memory.

junioraguilar commented 2 months ago

Do you use the exact model and .yaml provided by meta? Or change some parameters?

mvoodarla commented 2 months ago

no change in parameters! the exact model is whats being used.

chayryali commented 2 months ago

Thanks for sharing @mvoodarla! Indeed, there is a lot of room for optimization - looking forward to your blog post!

JunMa11 commented 2 months ago

Hi @mvoodarla ， thanks for sharing!

Any updates on the follow-up blog?