search

haruishi43 / equilib

🌎→🗾Equirectangular (360/panoramic) image processing library for Python with minimal dependencies only using Numpy and PyTorch
Apache License 2.0
156 stars 21 forks source link
360-degree camera cubemap-to-equirectangular equirectangular-images equirectangular-panorama equirectangular-projection numpy panorama perspective-projection python pytorch vr

readme

equilib

Processing Equirectangular Images with Python

PyPI version GitHub license
equilib

If you found this module helpful to your project, please site this repository:

@software{pyequilib2021github,
  author = {Haruya Ishikawa},
  title = {PyEquilib: Processing Equirectangular Images with Python},
  url = {http://github.com/haruishi43/equilib},
  version = {0.5.0},
  year = {2021},
}

Installation:

Prerequisites:

pip install pyequilib

For developing, use:

git clone --recursive https://github.com/haruishi43/equilib.git
cd equilib

pip install -r requirements.txt

pip install -e .
# or
python setup.py develop

NOTE: might not work for PyTorch>=2.0. If you have any issues, please open an issue.

Basic Usage:

equilib has different transforms of equirectangular (or cubemap) images (note each transform has class and func APIs):

There are no real differences in class or func APIs:

Each API automatically detects the input type (numpy.ndarray or torch.Tensor), and outputs are the same type.

The arguments for each class or func depends on the transform, but here are the common arguments:

An example for Equi2Pers/equi2pers:

Equi2Pers
 
equi2pers
 

```Python
import numpy as np
from PIL import Image
from equilib import Equi2Pers

# Input equirectangular image
equi_img = Image.open("./some_image.jpg")
equi_img = np.asarray(equi_img)
equi_img = np.transpose(equi_img, (2, 0, 1))

# rotations
rots = {
    'roll': 0.,
    'pitch': np.pi/4,  # rotate vertical
    'yaw': np.pi/4,  # rotate horizontal
}

# Intialize equi2pers
equi2pers = Equi2Pers(
    height=480,
    width=640,
    fov_x=90.0,
    mode="bilinear",
)

# obtain perspective image
pers_img = equi2pers(
    equi=equi_img,
    rots=rots,
)
```



```Python
import numpy as np
from PIL import Image
from equilib import equi2pers

# Input equirectangular image
equi_img = Image.open("./some_image.jpg")
equi_img = np.asarray(equi_img)
equi_img = np.transpose(equi_img, (2, 0, 1))

# rotations
rots = {
    'roll': 0.,
    'pitch': np.pi/4,  # rotate vertical
    'yaw': np.pi/4,  # rotate horizontal
}

# Run equi2pers
pers_img = equi2pers(
    equi=equi_img,
    rots=rots,
    height=480,
    width=640,
    fov_x=90.0,
    mode="bilinear",
)
```

For more information about how each APIs work, take a look in .readme or go through example codes in the tests or scripts.

Coordinate System:

Right-handed rule XYZ global coordinate system. x-axis faces forward and z-axis faces up.

You can chnage the right-handed coordinate system so that the z-axis faces down by adding z_down=True as a parameter.

See demo scripts under scripts.

Grid Sampling

To process equirectangular images fast, whether to crop perspective images from the equirectangular image, the library takes advantage of grid sampling techniques. Some sampling techniques are already implemented, such as scipy.ndimage.map_coordiantes and cv2.remap. This project's goal was to reduce these dependencies and use cuda and batch processing with torch and c++ for a faster processing of equirectangular images. There were not many projects online for these purposes. In this library, we implement varieties of methods using c++, numpy, and torch. This part of the code needs cuda acceleration because grid sampling is parallelizable. For torch, the built-in torch.nn.functional.grid_sample function is very fast and reliable. I have implemented a pure torch implementation of grid_sample which is very customizable (might not be fast as the native function). For numpy, I have implemented grid sampling methods that are faster than scipy and more robust than cv2.remap. Just like with this implementation of torch, numpy implementation is just as customizable. It is also possible to pass the scipy and cv2's grid sampling function through the use of override_func argument in grid_sample. Developing faster approaches and c++ methods are WIP. See here for more info on implementations.

Some notes:

Develop:

Test files for equilib are included under tests.

Running tests:

pytest tests

Note that I have added codes to benchmark every step of the process so that it is possible to optimize the code. If you find there are optimal ways of the implementation or bugs, all pull requests and issues are welcome.

Check CONTRIBUTING.md for more information

TODO:

Acknowledgements: