Jamy-L / Handheld-Multi-Frame-Super-Resolution

Handheld Multi-image Super-resolution [Wronski et al., SIGGRAPH19]. Non-official GPU-supported Python implementation.
MIT License
131 stars 20 forks source link

Artifacts in output #33

Open y2k0999 opened 11 months ago

y2k0999 commented 11 months ago

I'm getting weird artifacts in very fine parts of my images in certain color channels. Any idea what part of the model I can play with to try to resolve these artifacts?

Screenshot from 2023-12-12 18-22-24

y2k0999 commented 11 months ago

When I use less frames from my burst as input the effects get even more serious Screenshot from 2023-12-12 18-41-39

Jamy-L commented 11 months ago

We observed that these color artefacts were typically appearing when there are not enough samples during the local fusion. This can be because there are not enough frames (as you pointed out), but also because the fusion kernel is too thin, or because the alignment module did not perform good enough.

Your first picture is actually very interesting, because we can see it happening on the left: as the structure gets thinner, the fusion kernel becomes too thin and cant get enough samples. When the structure get too thin, the structure tensor can no longer pick up the high frequencies, so the fusion kernel gets larger and the artefact no longer appear.

To solve the problem, you can use this postprocessing that worked quite well for removing this type of artefacts. You can also increase k_detail, which should ensure that every colour channel gets enough samples during fusion.

If you really want to go deeper into the kernel engineering, the kernel shaping is done here. By playing around with different functions to drive k[0] and k[1] more or less agressively, one may get better results.