CPU profile 2 process throwing CUDA out of memory for one image with multiple items when propagate button is clicked

hkchengrex / MiVOS

[CVPR 2021] Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion. Semi-supervised VOS as well!

https://hkchengrex.com/MiVOS/

MIT License

464 stars 65 forks source link

CPU profile 2 process throwing CUDA out of memory for one image with multiple items when propagate button is clicked #27

Open zdhernandez opened 2 years ago

zdhernandez commented 2 years ago

@hkchengrex To replicate:

load only one image of width(3024 by 4032) in folder./example/test_folder/
run command: python interactive_gui.py --mem_profile 2 --images ./example/test_folder/ --resolution -1 --num_objects 4
click on one object to create overlay of the first object (red)
select num keypad 2 and click a different object (to produce overlay of different color)
select num keypag 3 and click a different object (to produce overlay of different color)
select num keypag 3 and click a different object (to produce overlay of different color)
click "propagate" Throws error. See picture.

Even though I am doing one image if I click "Save" it does what is supposed to to (save overlay and mask). But clicking "Propagate" should not throw and error with cuda when --mem_profile was set to 2, right ? should not have used GPU.

hkchengrex commented 2 years ago

I did not write the boundary condition for a single image, so it might be doing some extra work. It is also not the intended usage of the GUI. If you want single image segmentation you can just use the original f-BRS GUI: https://github.com/saic-vul/fbrs_interactive_segmentation

zdhernandez commented 2 years ago

@hkchengrex can you add something to protect the boundary condition when it's a single image so that the GUI won't crash, please? Also, it's trying to pull GPU resources but the memory profile given was of type 2. I was not expecting a CUDA out-of-memory error to appear.

hkchengrex commented 2 years ago

It still uses the GPU even with mem_profile=2. We just buffer less on the GPU. I cannot think of a general use case where it will be useful to do propagation on a single image. If you can create a pull request for that I can probably merge it.

zdhernandez commented 2 years ago

@hkchengrex Is just protection so that it won't crash. And yes, I think it's doing extra work. It'll take me forever to set the project in debug mode and add the changes. I would prefer it if you add the protection. That's probably the last thing that's making the GUI crash right now. I've tried other combinations and it's doing pretty well.

hkchengrex commented 2 years ago

If you don't need to propagate then just don't click that button

zdhernandez commented 2 years ago

@hkchengrex Nah this is more from a software test context. The GUI should never crash.

hkchengrex commented 2 years ago

I see. That's a memory error (it is expected to use the GPU) and if we were to solve it we would need to guard against all CUDA OOM errors. I can see how that's useful in a production setting but it is of very low priority in this for-research codebase. I agree that adding a guard is beneficial so pull requests are welcome.