Mask features

[piercus]

@lukasalexanderweber thanks for this library, very useful to play with sticthing

I need to be able to mask some features in the stitching process so some region in the image are ignored in the sticthing process.

I found other people on the web having the same needs :

• https://answers.opencv.org/question/192008/how-to-stitch-images-by-opencv-with-masks-only-stitch-parts-of-images/
• https://stackoverflow.com/questions/73721536/how-to-stitch-images-by-opencvsharp-using-mask

I have implemented this feature in command line.

Currently i have been able to make it work like :

python stitching/cli/stitch.py img1.png img2.png --mask_names mask1.png mask2.png --no-crop --blender_type no --finder no --compensator no --affine

This is a first implem and more tests need to be done, but i would like to have an early feedback as if you are open to integrate this new mask_feature capability

What do you think ?

[lukasalexanderweber]

Am I wrong or are your FeatureDetector changes missing?

[piercus]

@lukasalexanderweber It's not needed since *args pass the mask directly to computeImageFeatures2 in

    def detect_features(self, img, *args, **kwargs):
        return cv.detail.computeImageFeatures2(self.detector, img, *args, **kwargs)

I have added the mask to the warper in today's last commit (709906c)

[lukasalexanderweber]

ahh nice I didn't knew that this functionality is already covered by OpenCV using the optional parameter mask

I thought you want to implement it in python ontop.

Thanks for the feedback and for working on it :)

[piercus]

@lukasalexanderweber in the last version i'm only wrapping the non-masked parts of the image.

This is a design choice and this can be discussed, but eventually, the question is to use feature_mask in the wrapping process as it is done in

https://github.com/lukasalexanderweber/stitching/blob/709906c4f7a66a08064eaf559813b9478dfe363c/stitching/warper.py#L66

On my use case, it makes total sense to do a wrap-masking I'm not sure if other use cases would need feature-only masking (without wrap-masking)

[lukasalexanderweber]

On my use case, it makes total sense to do a wrap-masking

I don't quite get what you can achieve with the warped feature masks?

[piercus]

The use case is typically background stitching with foreground mask.

Credits: image come from https://github.com/BGU-CS-VIL/DeepMCBM/tree/main/input/tennis/frames

Input

First image

First Mask

Second image

Second mask

Command

python stitching/cli/stitch.py  frame00000.jpg frame00012.jpg --mask_names  mask00000.jpg mask00012.jpg --no-crop  --blender_type no  --finder no  --compensator no  --affine  --output tennis.png

Result with wrapping

Result without wrapping

[lukasalexanderweber]

https://github.com/BGU-CS-VIL/DeepMCBM#results -> wow this looks like magic

So to summarize:

• the user can pass in masks to specify in which regions features should be searched
• additionally, the user can have another option to let the masks be foreground mask for background stitching

[piercus]

the user can pass in masks to specify in which regions features should be searched

Yes

additionally, the user can have another option to let the masks be foreground mask for background stitching

Do you have an example in mind where we don't want to use the feature mask in the warping ?

In my use case (foreground masking), i'm masking parts of the image which won't match between images, if those part won't match, keeping them or not in the final result will be ambiguous and will depend on the order of the images.

[lukasalexanderweber]

Do you have an example in mind where we don't want to use the feature mask in the warping ?

yeah like when you have a thing in an image which has a lot of features but is technically not part of the image and leads to wrong matches. I had the case on scans where somehow barcodes were placed ontop ob the plan to scan and often the barcodes where matched instead of the plan content. This feature would have helped me a lot.

[lukasalexanderweber]

I worked on it a while yesterday evening and thought that the proposed changes have the potential to bloat the standard stitching workflow. I thought that it might be possible creating some kind of BackgroundStitcher which extends Stitcher. What do you think of that? In the CLI, if feature masks are specified, we will then create an instance of BackgroundStitcher

[piercus]

@lukasalexanderweber ok !

Can you please comment the PR or make a PR over my PR with your changes so I can try to help you back in this ?

I'm afraid to work in parallel on different approaches.

Thanks

[lukasalexanderweber]

Hello @piercus!

I want to check back on you if you've solved the issue for you and if you still have interest in contributing? I was busy improving the testability of the package so I should be able to help you faster now. In https://github.com/OpenStitching/stitching/issues/94 only the first part of masking features is needed, independently of background stitching (I guess)

[piercus]

@lukasalexanderweber I feel we should keep it as simple as possible and maybe only focus on the first part of the masking features

[3nol]

I just came across related issues regarding passing a mask to the feature detector. I am very interested in having this feature in the stitching package but I noticed that this PR is open for some time.

Since no further progress has been made here, I opened another PR#104 to finish it.

[lukasalexanderweber]

continued in https://github.com/OpenStitching/stitching/pull/104