New Adapter: dual fish eyes

[truongsinh]

Describe the feature

Insta360 and Gear 360 save their raw image as dual fish eye texture. I have Insta360 and can provide some of their video/picture for reference. Insta360 provide studio app on Mac and Windows with different stitch strategy, but I think we can just get 1 to work first, and can improve later.

Alternatives you've considered

Manually convert all Insta360 image and video to equirectangular

Additional context

Stitching options from Insta360 studio

Prior art: https://github.com/acalcutt/Gear360_html5_viewer

No response

[jtoppine]

Just adding that Ricoh Theta Z1 (one of the most popular "pro" cameras for specifically stills) raw images are also dual fisheye, DNG. (JPG's are equirectangular, and raw images contain an embedded full size preview or "thumbnail" image, which is confusingly equirectangular unlike the dual fisheye main content). I can provide Ricoh files if needed.

Having PSV support this projection would be interesting, as it would simplify workflows which do not, or can not use the manufacturer's software (eg. on linux, or web servers with transcoding capabilities). One could edit/resize/adjust the files directly in generic raw software and serve direct to PSV, without the Ricoh apps or compatible stitching suites.

[mistic100]

yes please share some examples

[jtoppine]

Not trying to hijack OP's feature request here; if I understood correctly this is identical feature/usecase, just a different camera. I have no experience with Insta360 cameras or files, but I imagine they use similar approach. Anyway:

Here is a set of two sample files from Ricoh Theta Z1. Even one raw file from the camera is too large for github, so sharing via dropbox: https://www.dropbox.com/scl/fi/5vx0ood4avhckgy3zlvwd/psv_ricoh_theta_z1_samplefiles.zip?rlkey=pkwjjmpcpq9cdxolk4p8resg0&dl=0

Two sample files, for both:

• raw file direct from camera (dual fisheye raw data + equirectangular preview)
• jpg file direct from camera (equirectangular, which of course works well with PSV already)
• dual fisheye jpg made from the raw file with darktable (unfortunately, I'm unsure how to retain or create the proper panorama projection metadata so that a theoretical dual fisheye -compatible viewer could detect the image as such and be able to view it properly)
• exif metadata from the raw file as text, as reported by exiftool, for convenience

Here's how I imagine the workflow to go if PSV would support dual fisheye projection:

1. users store/upload raw files to a web server
2. server processes the raw file with some parameters/adjustments, and converts it to a browser compatible format. this would naturally be dual fisheye as that's what the raw is
3. (server app would probably need to insert the proper projection related EXIF tags, or otherwise let PSV know it is indeed a dual fisheye projection image)
4. PSV is used to view the processed files

[truongsinh]

Not trying to hijack OP's feature request here; if I understood correctly this is identical feature/usecase, just a different camera. I have no experience with Insta360 cameras or files, but I imagine they use similar approach. Anyway:

Correct, same format, just different cam :)

[mistic100]

So I tried to adapt the linked code to newer versions of ThreeJS. The fact is I don't understand the math involved, but my version does not work, with or without the updates of UVs, the result is the same: there a big seam and deformation on the junction.

import { BufferAttribute, BufferGeometry, Material, Mesh, MeshBasicMaterial, SphereGeometry } from 'three';
import { type Viewer } from '../Viewer';
import { EquirectangularAdapter } from './EquirectangularAdapter';
import { SPHERE_RADIUS } from '../data/constants';

export class DualFisheyeAdapter extends EquirectangularAdapter {
    static override readonly id: string = 'dual-fisheye';
    static override readonly VERSION = PKG_VERSION;

    constructor(viewer: Viewer) {
        super(viewer);
    }

    override createMesh(): Mesh<BufferGeometry, Material> {
        const geometry = new SphereGeometry(
            SPHERE_RADIUS,
            this.SPHERE_SEGMENTS,
            this.SPHERE_HORIZONTAL_SEGMENTS
        ).scale(-1, 1, 1).toNonIndexed();

        const uvs = geometry.getAttribute('uv') as BufferAttribute;
        const normals = geometry.getAttribute('normal') as BufferAttribute;

        for (let i = 0; i < uvs.count; i++) {
            for (let j = 0; j < 3; j++) {
                const x = normals.getX(j);
                const y = normals.getY(j);
                const z = normals.getZ(j);

                if (i < uvs.count / 2) {
                    const correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
                    uvs.setXY(j,
                        x * (1 / 4) * correction + (1 / 4),
                        z * (1 / 2) * correction + (1 / 2)
                    );
                } else {
                    const correction = (x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
                    uvs.setXY(j,
                        -1 * x * (1 / 4) * correction + (3 / 4),
                        z * (1 / 2) * correction + (1 / 2)
                    );
                }
            }
        }

        return new Mesh(geometry, new MeshBasicMaterial());
    }
}

[mistic100]

The adaptation was wrong, this one works much better. There is magic constant c = 0.947 which might depend on the specific shot/lens...

import { BufferAttribute, BufferGeometry, Material, Mesh, MeshBasicMaterial, SphereGeometry } from 'three';
import { type Viewer } from '../Viewer';
import { SPHERE_RADIUS } from '../data/constants';
import { EquirectangularAdapter } from './EquirectangularAdapter';

export class DualFisheyeAdapter extends EquirectangularAdapter {
    static override readonly id: string = 'dual-fisheye';
    static override readonly VERSION = PKG_VERSION;

    constructor(viewer: Viewer) {
        super(viewer);
    }

    override createMesh(): Mesh<BufferGeometry, Material> {
        const geometry = new SphereGeometry(
            SPHERE_RADIUS,
            this.SPHERE_SEGMENTS,
            this.SPHERE_HORIZONTAL_SEGMENTS
        ).scale(-1, 1, 1).toNonIndexed();

        const uvs = geometry.getAttribute('uv') as BufferAttribute;
        const normals = geometry.getAttribute('normal') as BufferAttribute;

        for (let i = 0; i < uvs.count; i++) {
            for (let j = 0; j < 3; j++) {
                const index = i * 3 + j;

                const x = normals.getX(index);
                const y = normals.getY(index);
                const z = normals.getZ(index);

                const c = 0.947;
                if (i < uvs.count / 6) {
                    const correction = (x == 0 && z == 0) ? 1 : (Math.acos(y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
                    uvs.setXY(index,
                        x * (c / 4) * correction + (1 / 4),
                        z * (c / 2) * correction + (1 / 2)
                    );
                } else {
                    const correction = (x == 0 && z == 0) ? 1 : (Math.acos(-y) / Math.sqrt(x * x + z * z)) * (2 / Math.PI);
                    uvs.setXY(index,
                        -1 * x * (c / 4) * correction + (3 / 4),
                        z * (c / 2) * correction + (1 / 2)
                    );
                }
            }
        }

        geometry.rotateX(-Math.PI / 2);
        geometry.rotateY(Math.PI);

        return new Mesh(geometry, new MeshBasicMaterial());
    }
}

There is still a visible seam, which I was still not able to hide, even with the other magic numbers on this version https://github.com/acalcutt/Gear360_html5_viewer/blob/master/lib/360-view-image.js

[mistic100]

@truongsinh @jtoppine do you have an example panorama I could use for a demo on PSV website ?
preferably less dark than the warehouse and less creepy than the bathroom :)

[jtoppine]

@truongsinh @jtoppine do you have an example panorama I could use for a demo on PSV website ? preferably less dark than the warehouse and less creepy than the bathroom :)

Hahah :) I don't have many fitting samples as my images are mostly funder NDA or have faces and stuff I'd need to clear. But here's one, less creepy image that might fit your needs: https://toppinen.net/projects/psv/dualfisheye-demo.zip It is from Kotka archipelago, Finland.

This image is actually taken by Jonna Luostari https://jonnaluostari.com/ who is here beside me and gives permission to use it on the site. She's happy to be credited but it's totally ok also without a mention, no stress!

[Rai-Rai]

hi @truongsinh, not sure if I missed something here. But if you take a 360 deg photo (insta360 x3) you normally choose either jpg+raw or pure shot to get the best result. And then you have to process the files with the studio to apply the pureShot effect. When exporting you end up with a equirectangular panorama. As far as I know there's no way to do that directly on the device.

[mistic100]

Can someone answer Rai-Rai question ? To make sure adding this adapter is really useful.

[jtoppine]

Well, my use case may very well be a niche case. But here's why I think the feature would be at least a bit useful:

I have an archive of 360 photos, mostly from Ricoh Theta Z1 but potentially from other cameras too. Some are JPG (equirectangular), some RAW (DNG, dual fisheye), some have both, JPG+RAW.

Firstly, I might do manual editing/photoshopping/color correction of some files. That would happen preferably from the dual fisheye raw files as a source, and I would prefer to not need a stitching software to reproject those when saving them to JPG or maybe TIFF. So those edited files would be dual fisheye still. Working from Linux, official camera softwares are not available for stitching, so reprojection to equirectangular can be a bit of a pain.

Then I have a web app that acts as a file browser / gallery / viewer for said files. The frontend uses PSV to display panoramas. It would be nice to support many file formats from many different cameras without official camera software suites (which probably can't be installed on linux servers anyway).

It's pretty easy to convert any files including raw files server side to a web compatible format for display in a browser client. But if I also have to reproject the files from dual fisheye to equirectangular, server-side, that's more tricky (although probably possible with hugin or something). Would be kinda nice if PSV could support either projection as-is.

[rishubil]

I'd like to share my use case here.

I travel and shoot 360-degree videos with an Insta360. The original videos are stored as-is on my personal NAS. I could convert the videos beforehand, but since I don't know which ones I'll end up using, I just keep them on the NAS in their original state.

Sometimes I want to look back on a trip later or show a video to others. But there's no way to tell which of the original videos in the NAS's web-based file browser is the one I want. So I wanted a web-based 360-degree video player that could quickly identify the contents of the files.

However, a few days ago, I found this project and based on the contents of the dual-fisheye branch, I roughly created a DualFisheyeVideoAdapter and tested it.

The DualFisheyeVideoAdapter can read and play .lrv files, so I can quickly find the videos I want on the NAS. (Videos shot with the Insta360 X3 are saved as low-resolution dual-fisheye MP4 files with the .lrv extension, and as two separate high-resolution videos with the .insv extension).

It would be nice if it could also read and play .insv files, but as it is, I'm very happy with it.

[mistic100]

The library will never be able to read obscure proprietary formats, it entirely relies on the browser decoding capability. Also I will not be able to read and synchronize two video files, they will have to be glued in a single file beforehand, if the low-resolution version is not HQ enough.

[rishubil]

The library will never be able to read obscure proprietary formats, it entirely relies on the browser decoding capability. Also I will not be able to read and synchronize two video files, they will have to be glued in a single file beforehand, if the low-resolution version is not HQ enough.

I understand. What I meant to say was that the implementation of the DualFisheyeAdapter was very useful for my actual use case.

[github-actions[bot]]

This feature/bug fix has been released in version 5.8.0.