search

coolzhao / Geo-SAM

A QGIS plugin tool using Segment Anything Model (SAM) to accelerate segmenting or delineating landforms in geospatial raster images.
MIT License
199 stars 26 forks source link

Discussion : Image Encoding #16

Open shizurumaya opened 10 months ago

shizurumaya commented 10 months ago

Hello,

I noticed that the checkpoint you're using is only 15 MB in size, while the meta checkpoints referenced here (https://github.com/facebookresearch/segment-anything#model-checkpoints) are over 2 GB. Could you please clarify the reason for this discrepancy?

Thank you in advance for your insights.

Fanchengyan commented 10 months ago

Hi @shizurumaya . This checkpoint is not from the original SAM. In order to allow users to interact in real time to use SAM to segment objects, we modified the SAM workflow (see https://geo-sam.readthedocs.io/en/latest/#introduction). This checkpoint was created by us to avoid having to encode the image each time new prompts are provided.

shizurumaya commented 10 months ago

Thank you for the response. This tool seems very promising. It's already working well on my end :)

The size of the elements seems to be limited by the tile size. For instance, when I select a large forest, I get a square tile. Is there any plan for the tool to be able to outline elements that are larger than the patch size?

Additionally, I see that the size limit during encoding is 1024. Would the segmented elements potentially be larger if this limit was increased to 2048? forest example

Fanchengyan commented 10 months ago

Hi @shizurumaya , you have the option to adjust the resolution while encoding the image. Additionally, we have developed a copilot tool to assist users in determining the appropriate resolution.To access this tool, please download the pre-release version. For more detailed information, please refer to the following link: Encoder Copilot Tool (QGIS).

Fanchengyan commented 10 months ago

Currently, SAM only supports image inputs with a resolution of 1024x1024 pixels. As mentioned in my previous comment, reducing the image resolution is a viable solution for most scenarios. However, it seems that changing the resolution may still not cover the entire forest in your specific use case. We are aware of this limitation, which also applies to delineating large glacial areas. We are actively exploring potential solutions to address this issue, but we have not found a satisfactory approach yet. One possible but cumbersome workaround is to manually delineate and save only the boundary portions, then merge these polygons and extract the outer boundary line to achieve the desired extraction. However, this approach poses several challenges and may not provide the best user experience. It is indeed a challenging problem that requires further consideration.

4del-Yousefi commented 10 months ago

Hi @Fanchengyan, can you check https://github.com/opengeos/segment-geospatial? I think you might find a solution for 1024x1024 within their code. it's not a QGIS plugin but it works on tif images and might be useful for you.

Fanchengyan commented 10 months ago

Hi @4del-Yousefi . Their work also requires resizing the original image. We now have a preliminary idea for solving extremely large objects, and have added it to the Future Works section.