Numpy cube2equi does not consider 'mode' and uses bilinear interpolation instead

[gerhardneuhold]

Thanks for this great library.

While converting cube maps to equirectangular (cube2equi) with numpy, I noticed that nearest neighbor interpolation is not used if mode is set to nearest.

The issue belongs to the following line, where always bilinear interpolation is used: https://github.com/haruishi43/equilib/blob/4297a6750593a3b29c88e296343a74283c2315e3/equilib/cube2equi/numpy.py#L317-L320

Work around for me is to set overrice_func to numpy_grid_sample as following:

    from equilib import cube2equi
    from equilib.grid_sample import numpy_grid_sample

    image_cube = ...
    h, w = ...  

    mode = "nearest"

    image_equi = cube2equi(
        cubemap=image_cube,
        cube_format="dice",
        width=w,
        height=h,
        clip_output=True,
        mode=mode,
        override_func=numpy_grid_sample,
    )

While this work around works for me, it is not intuitive to use. Will it be possible to support nearest neighbor interpolation as expected by setting mode=nearest or is there a specific reason why it is done the way it is now?

[haruishi43]

@gerhardneuhold thanks for raising the issue. The explanation will be long so the tldr is: I currently don't support nearest neighbor interpolation for cube2equi because the current implementation causes weird interpolations between cube boundaries.

The general implementation (from equilib.grid_sample import numpy_grid_sample) does not handle interpolation across multiple images which is required in cube2equi transformation. The numpy_grid_sample inside equilibrium/cube2equi/numpy.py was implemented to mitigate interpolation artifacts between cube boundaries (where the cube maps stitch). Input cubemap: __General interpolation implemented in grid_sample: Notice that there are artifacts near the cube map boundaries. Improved interpolation__ (the current bilinear interpolation):

The improved interpolation is not efficient right now since I have to use multiple for-loops and I would like to improve this when I have the time.

[gerhardneuhold]

Thanks for the very good explanation @haruishi43 👍 What do you think about adding an assertion if "nearest" is used in cube2equi to let others know it is not working at the moment?

[haruishi43]

@gerhardneuhold I think that's a great idea. You could send me a pull request if you already implemented it, if not I will add it later

[gerhardneuhold]

@haruishi43 I don't have it implemented yet. Would be great if you can add it later. Thanks.