search

SHI-Labs / OneFormer

OneFormer: One Transformer to Rule Universal Image Segmentation, arxiv 2022 / CVPR 2023
https://praeclarumjj3.github.io/oneformer
MIT License
1.39k stars 129 forks source link

out of memory when using sem seg post processing before inference #87

Open TianjinTeda opened 10 months ago

TianjinTeda commented 10 months ago

Hi,

I am recently using your model and I think it was an excellent work!

However, I noticed that you set the value sem_seg_postprocess_before_inference to be true when panoptic_on or instance_on is true. Is it mandatory to do that? I am using the same setting and it keeps reporting oom when doing the inference on relatively larger images and the computing are sent to cpu which leads to very low speed.

Looking forward to hearing from you, thank you in advance!

praeclarumjj3 commented 10 months ago

Hi @TianjinTeda, the sem_seg_postprocess_before_inference is crucial because we check that flag to decide whether to resize the predictions to the original resolution before post-processing. If your image resolution is very high compared to your GPU memory, it's expected to obtain an OOM error.

https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/oneformer_model.py#L327-L330

If you want to try something different, you could resize the final panoptic_map and result.pred_masks instead. However, note that we have not tested the performance using this approach. https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/oneformer_model.py#L434 https://github.com/SHI-Labs/OneFormer/blob/4962ef6a96ffb76a76771bfa3e8b3587f209752b/oneformer/oneformer_model.py#L475