search

uncbiag / SimpleClick

SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (ICCV 2023)
MIT License
209 stars 32 forks source link

Is it very time-consuming to evaluate the model? #11

Closed songbingze closed 1 year ago

songbingze commented 1 year ago

I tried to evaluate the model on a remote sensing image dataset and found that testing 1300 512 by 512 sized images would take approximately 7 hours. Is this normal?

qinliuliuqin commented 1 year ago

It seems abnormal, but could you provide more context about your evaluation settings?

  1. Which model did you use? ViT-H, ViT-L, or ViT-B?
  2. Did you evaluate on CPU or GPU?
  3. How many objects are in each image? The speed is proportional to the total objects, not the images.
  4. Were there any other programs running on the machine during the evaluation? According to my experience, it took ~30mins for the ViT-B model to evaluate SBD (2857 images, 6671 instances) on an NVIDIA RTX A6000 GPU.
songbingze commented 1 year ago

Thank you so much for taking the time to reply to my message. I truly appreciate your help and expertise. After reading your response, I think I understand why the speed is so slow. It seems that each of my images contains around 20-40 instances, and labeling them accurately requires labeling 11+ points, which can be quite challenging due to the small, scattered, and complex edges, as well as the blurred nature of remote sensing images. I understand that this can significantly affect the evaluation speed. Once again, thank you for your valuable insights and assistance.

qinliuliuqin commented 1 year ago

Yes, that makes a lot of sense. Thanks for the question!