Where should I go for real-time instance segmentation.

[cozeybozey]

Prerequisite

• [X] I have searched the existing and past issues but cannot get the expected help.
• [X] I have read the FAQ documentation but cannot get the expected help.
• [X] The bug has not been fixed in the latest version.

💬 Describe the reimplementation questions

I am looking for real-time instance segmentation models and at first sight the openmmlab repositories seemed to be great for this. However I am now a little bit confused as to which repositories and models actually support instance segmentation. Obviously the models in mmsegmentation support instance segmentation however the models implemented there are not really what I am looking for. I am especially interested in RTMDet and the Yolo series. That is why I thought the mmdetection and mmyolo repositories might have what I needed, however am I correct in assuming that no models in mmyolo support instance segmentation? If I am wrong then can you tell me how to run instance segmentation? I believe there are models in mmdetection that support instance segmentation like mask_rcnn, however the RTMDet version there does not seem to support it. So I guess my questions is, is it currently possible for me to use real-time instance segmentation with a Yolo model or with RTMDet? If not, will it be supported in the future?

Environment

sys.platform: linux
Python: 3.8.16 (default, Mar  2 2023, 03:21:46) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0,1: NVIDIA GeForce GTX 1080
CUDA_HOME: /usr
NVCC: Cuda compilation tools, release 11.8, V11.8.89
GCC: gcc (Debian 12.2.0-14) 12.2.0
PyTorch: 1.10.1
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.11.2
OpenCV: 4.7.0
MMEngine: 0.7.0
MMCV: 2.0.0rc4
MMDetection: 3.0.0rc6
MMYOLO: 0.5.0+dc85144

Expected results

No response

Additional information

No response

[hhaAndroid]

@cozeybozey Currently supporting

[cozeybozey]

Sorry I don't really understand what this comment means

[Nioolek]

Hello, Support instance segmentation in mmyolo is in our codemap. MMYOLO has supported YOLOv8 and RTMDet instance segmentation inference in dev branch. And training will be supported in future release

[JKelle]

I'm also interested in using RTMDet instance segmentation. Do you have an estimate for when it will be released?

I found a config for it (configs/rtmdet/rtmdet-ins_s_syncbn_fast_8xb32-300e_coco.py) but I don't see where I can download the pre-trained model weights for rtmdet-ins_s.

[jimmy0502]

Hello, Support instance segmentation in mmyolo is in our codemap. MMYOLO has supported YOLOv8 and RTMDet instance segmentation inference in dev branch. And training will be supported in future release Hi Nioolek, I'm also looking for yolov8 instance segmentaion model. I change to dev branch, according to your suggestion, but I can't find any files with 'ins' in the filename in configs/yolov8 folder. Are the instance segmentation model config files in yolov8 folder? or somewhere else? thanks.