search

floe / backscrub

Virtual Video Device for Background Replacement with Deep Semantic Segmentation
Apache License 2.0
734 stars 85 forks source link

crash with model body-pix-float-050-8.tflite #151

Open jjsarton opened 2 years ago

jjsarton commented 2 years ago

System: Fedora 36

Backscrub: version heads/main-0-g4f70bc1 (Tensorflow: build tags/v2.8.0-0-g3f878cff5b6, run-time 2.8.0)

opencv 4.5.5

Launching backscrub with:

backscrub -c /devvideo0 -v /dev/video3 -m body-pix-float-050-8.tflite

crash at libbackscrub.cc line 368

    cv::resize(ctx.ofinal(ctx.in_roidim),tmpbuf,ctx.mroi.size());

ctx.ofinal has a size of 33x33 and ctx.in_roidim 257x257

Modify the code to

    ctx.ofinal.copyTo(tmpbuf);
        cv::resize(tmpbuf,tmpbuf,ctx.mroi.size());

solve the problem partially. Of course, the result is not very nice. (With the arguments -d -d there is a crash again).

BenBE commented 2 years ago

The -d increases the debug level. Thus if you see the crash with the more verbose code paths this indicates that you're still hitting some invalid states that just are hidden on low verbosity.

jjsarton commented 2 years ago

The effect of the debug level is known.

The primary problem is the call cv::resize(ctx.ofinal(ctx.in_roidim),tmpbuf,ctx.mroi.size());.

As far I understood the code, ctx.ofinal(ctx.in_roidim) implies scaling (here) or cropping. The call within cv::resize don't always work. I had a similar issue while trying to crop the input image in order to not have the video output stretched.

phlash commented 1 year ago

As we have now identifed here: https://github.com/floe/backscrub/pull/153#discussion_r967831025 this is a result of the output stride inherent in the model, that our existing design takes no account of.

jjsarton commented 1 year ago

For the body-pix segmentation we have an output size 33x33 and with my modifications I don't have seen differences between the original and the modified code (if I except crashes, which are no more present).