search

kunzmi / ImageStackAlignator

Implementation of Google's Handheld Multi-Frame Super-Resolution algorithm (from Pixel 3 and Pixel 4 camera)
GNU General Public License v3.0
392 stars 65 forks source link

Does this approach need the camera parameters? #21

Closed Remember2018 closed 3 years ago

Remember2018 commented 3 years ago

Hi @kunzmi , I found that in your code, there is a file named ExtraCameraProfiles.xml to get the camera profiles. Is this a necessity for this approach?

If we just have several images without the camera parameters, is this method still suitable for their multi-frame super-resolution?

kunzmi commented 3 years ago

You need the noise model parameters, otherwise the algorithm cannot guess if a patch is just noisy flat signal or some actual feature. Some DNG files already have these values stored, some don't, in which case you need the ExtraCameraProfiles.xml. If you open a file and some values appear for Noise model alpha you should be good.

Remember2018 commented 3 years ago

Hi, @kunzmi , thanks a lot for your kind comment! It is possible that first convert the multi-frame RGB images back to PEF files under some-certain camera model (e.g., the camera in your code), then adopt your code to conduct the multi-frame super-resolution? Given the multi-frame RGB images, their bayer pattern parameters are missing. So I'm not sure if this is a possible solution to use your code in the ordinal multi-frame RGB images for their super-resolution.

Currently I'm trying to use the TestMode in your code. And replace the randomly generated image in your code to a given RGB image (the RGB image is first converted to its grayscale image, and as the frame1 to fit the TestRawFile, the other frames are generated as shown in your code). The parameters are set as your noted in the README.md. But it seems the final result seems have problems.

If convenient, could you please help give some advice? Thanks.

frame1 frame2 frame3 frame4 frame5 pre-align patches2 patches3 patches4 patches5 accumulate

kunzmi commented 3 years ago

Hi, The original paper and this code are meant to debayer RAW images and additionally get super-resolution, whereas it is for sure possible to only use the super-resolution part, it is not usable as is without adaptions. So in case you already have all three color planes, you'd need to adapt the code and skip all debayering steps (and add file/image support for your images). In my eyes it doesn't make sense to try out super-resolution by first reducing the resolution by applying a bayer-pattern, so recreating a PEF file wouldn't make much sense here...

Remember2018 commented 3 years ago

Hi, @kunzmi , thanks a lot for your suggestion!