search

facebookresearch / segment-anything

The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
Apache License 2.0
47.18k stars 5.59k forks source link

Reproducing segmentation results shown in the demo #230

Open hkzhang-git opened 1 year ago

hkzhang-git commented 1 year ago

Here are the segmentation results obtained by using the auto-segmentation button provided on the official website (https://segment-anything.com/demo)

image image

These are the segmentation results generated using the released code. Specifically, I used the ViT-H model with all hyperparameters set to their default values. image image

The results generated by the released code are obviously poorer than those obtained from the official website. Why is this the case, and what model is used in the website version?

ANTONIOPSD commented 1 year ago

There are likely custom settings applied that may be found in the WebAssembly file located at: https://segment-anything.com/js/ort-wasm-simd.wasm.

Decompiler:

https://webassembly.github.io/wabt/demo/wasm2wat/index.html

Good Luck 😅

Found API?:

{
    "openapi": "3.0.1",
    "info": {
        "title": "RESTful API for: automatic_masks",
        "version": "1.0.0"
    },
    "paths": {
        "/prediction/automatic_masks": {
            "post": {
                "description": "A predict entry point for model: automatic_masks.",
                "operationId": "automatic_masks",
                "parameters": [],
                "responses": {
                    "200": {
                        "description": "OK"
                    },
                    "500": {
                        "description": "Internal Server Error",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "type": "object",
                                    "required": [
                                        "code",
                                        "type",
                                        "message"],
                                    "properties": {
                                        "code": {
                                            "type": "integer",
                                            "description": "Error code."
                                        },
                                        "type": {
                                            "type": "string",
                                            "description": "Error type."
                                        },
                                        "message": {
                                            "type": "string",
                                            "description": "Error message."
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        },
        "/v1/models/{model_name}:predict": {
            "post": {
                "description": "A predict entry point for model: automatic_masks.",
                "operationId": "automatic_masks",
                "parameters": [],
                "responses": {
                    "200": {
                        "description": "OK"
                    },
                    "500": {
                        "description": "Internal Server Error",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "type": "object",
                                    "required": [
                                        "code",
                                        "type",
                                        "message"],
                                    "properties": {
                                        "code": {
                                            "type": "integer",
                                            "description": "Error code."
                                        },
                                        "type": {
                                            "type": "string",
                                            "description": "Error type."
                                        },
                                        "message": {
                                            "type": "string",
                                            "description": "Error message."
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        },
        "/v2/models/{model_name}/infer": {
            "post": {
                "description": "A predict entry point for model: automatic_masks.",
                "operationId": "automatic_masks",
                "parameters": [],
                "responses": {
                    "200": {
                        "description": "OK"
                    },
                    "500": {
                        "description": "Internal Server Error",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "type": "object",
                                    "required": [
                                        "code",
                                        "type",
                                        "message"],
                                    "properties": {
                                        "code": {
                                            "type": "integer",
                                            "description": "Error code."
                                        },
                                        "type": {
                                            "type": "string",
                                            "description": "Error type."
                                        },
                                        "message": {
                                            "type": "string",
                                            "description": "Error message."
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
hkzhang-git commented 1 year ago

There are likely custom settings applied that may be found in the WebAssembly file located at: https://segment-anything.com/js/ort-wasm-simd.wasm.

Decompiler:

https://webassembly.github.io/wabt/demo/wasm2wat/index.html

Good Luck 😅

Found API?:

{
  "openapi": "3.0.1",
  "info": {
      "title": "RESTful API for: automatic_masks",
      "version": "1.0.0"
  },
  "paths": {
      "/prediction/automatic_masks": {
          "post": {
              "description": "A predict entry point for model: automatic_masks.",
              "operationId": "automatic_masks",
              "parameters": [],
              "responses": {
                  "200": {
                      "description": "OK"
                  },
                  "500": {
                      "description": "Internal Server Error",
                      "content": {
                          "application/json": {
                              "schema": {
                                  "type": "object",
                                  "required": [
                                      "code",
                                      "type",
                                      "message"],
                                  "properties": {
                                      "code": {
                                          "type": "integer",
                                          "description": "Error code."
                                      },
                                      "type": {
                                          "type": "string",
                                          "description": "Error type."
                                      },
                                      "message": {
                                          "type": "string",
                                          "description": "Error message."
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
      },
      "/v1/models/{model_name}:predict": {
          "post": {
              "description": "A predict entry point for model: automatic_masks.",
              "operationId": "automatic_masks",
              "parameters": [],
              "responses": {
                  "200": {
                      "description": "OK"
                  },
                  "500": {
                      "description": "Internal Server Error",
                      "content": {
                          "application/json": {
                              "schema": {
                                  "type": "object",
                                  "required": [
                                      "code",
                                      "type",
                                      "message"],
                                  "properties": {
                                      "code": {
                                          "type": "integer",
                                          "description": "Error code."
                                      },
                                      "type": {
                                          "type": "string",
                                          "description": "Error type."
                                      },
                                      "message": {
                                          "type": "string",
                                          "description": "Error message."
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
      },
      "/v2/models/{model_name}/infer": {
          "post": {
              "description": "A predict entry point for model: automatic_masks.",
              "operationId": "automatic_masks",
              "parameters": [],
              "responses": {
                  "200": {
                      "description": "OK"
                  },
                  "500": {
                      "description": "Internal Server Error",
                      "content": {
                          "application/json": {
                              "schema": {
                                  "type": "object",
                                  "required": [
                                      "code",
                                      "type",
                                      "message"],
                                  "properties": {
                                      "code": {
                                          "type": "integer",
                                          "description": "Error code."
                                      },
                                      "type": {
                                          "type": "string",
                                          "description": "Error type."
                                      },
                                      "message": {
                                          "type": "string",
                                          "description": "Error message."
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
      }
  }
}

Thank you for your response. I followed your suggestion and attempted to locate the custom settings in the WebAssembly file. Unfortunately, I was unable to obtain the information necessary to replicate the segmentation results.

OmerMachluf commented 1 year ago

I'm not sure the issue is within the actual web assembly file, I think the problem is with the backbone model. see the issue i opened here: https://github.com/facebookresearch/segment-anything/issues/224 From what i'm seeing, using the embedding file retrieved from their backbone model gives much better result than the backbone model im hosting locally.

shbkukuk commented 1 year ago

i am following this issue

mahmoudtabikh commented 1 year ago

I tried using the 2 other as models as well changing some of their parameters and i still get this issue. especially for detecting beards on humans.

sssmallmonster commented 1 year ago

I found this problem also. And I used the Hover & Click button to make multiple prompts, but I found the results generated by the released code are obviously poorer than those obtained from the official website demo. I suspect that the model weight of the demo on the official website is different from the weight of the open code, that is to say, the weight of the web demo is not ViT-H.