Add magic wand with Facebooks Segment Anything Model (SAM) to select OSM-Features

[tordans]

Description

https://twitter.com/picterra/status/1646887895519641602 has a great example of how SAM can be used to select features based on aerial imagery.

It would be great to have SAM in RapID to select features which then get auto-matched to Presets (when possible).

[bhousel]

Thanks! I agree it would be very fun and useful to add a magic wand tool. We might not actually need to (or be allowed to) send this data out to Segment Anything and get a result back with the polygon - I think the simpler path might be to build a tool like this with the pixel data we have already in the imagery.

[tordans]

For reference: This blog post https://www.openstreetmap.org/user/Kshitijraj%20Sharma/diary/401699 looks into using SAM for OSM.

I still struggle to see how a mapping workflow would be improve by this tooling. On the one hand those processes help selecting things or more precisely they create shapes, which one could use a magic wand tool for instead. On the other hand, SAM also categorizes stuff, so I could open a region and do the equivalent of Photoshop "select all red color" by saying "select all trees in this image" and SAM would create notes with OSM tree tags for me to review (accept or reject) and then add to OSM. In this sense, SAM could be an inline version of the external dataset selection that Rapid has right now.

[kshitijrajsharma]

I agree with this idea, It would be useful for mappers to have a option like magic tool, we don't need to push back to SAM, as SAM can run on browser this could be a good option for mappers ! We don't need to categorize it, we can simply use it to trace and the rest of mapping workflow remains the same ! Happy to volunteer !!

[tordans]

FYI, there is a 5 Min video from Mapping USA 2024 at https://www.youtube.com/watch?v=FsCInkvB7gM which shows how to use the segment anything model with the Magic wand plugin in JOSM (https://github.com/developmentseed/JosmMagicWand). This does use a backend to segment the data.

[lectrician1]

Also, I swear, this should be the most important engineering project for OSM right now. The editing scale for this is endless. I would love to work on this, but I don't have the time, like many people. The EWG should find someone to work on it and pay them.