Stereo 3D Rendering Modes

[AndrewHazelden]

When the --rendering-mode 1 CLI flag is enabled the built-in "Anaglyph" stereo 3D rendering option is enabled.

It would be nice to have support for several different stereo 3D rendering modes. This would allow passive stereo 3D displays to be used when connected with an "extended desktop" monitor output mode in the display preferences.

Additional rendering modes like SBS (Side by Side), OU (Over/Under), and Interlace stereo 3D output would be possible with the addition of several extra fragment shaders in the "shaders\core\" folder.

Stereo SBS Output Example

Here is a patched version of the "anaglyph.fp" file that supports Side By Side stereo3D output:

/*
 * Copyright (C) 2020, Inria
 * GRAPHDECO research group, https://team.inria.fr/graphdeco
 * All rights reserved.
 *
 * This software is free for non-commercial, research and evaluation use 
 * under the terms of the LICENSE.md file.
 *
 * For inquiries contact sibr@inria.fr and/or George.Drettakis@inria.fr
 */

#version 420

layout(binding = 0) uniform sampler2D left;
layout(binding = 1) uniform sampler2D right;
layout(location= 0) out vec4 out_color;

in vec2 vertex_coord;

void main(void) {
    vec2 texcoord = (vertex_coord + vec2(1.0)) / 2.0;
    vec2 tl = vec2(texcoord.x / 0.5, texcoord.y);
    vec2 tr = vec2(texcoord.x / 0.5, texcoord.y);
    vec4 cl = texture(left, tl);
    vec4 cr = texture(right, tr);
    out_color = vertex_coord.x < 0 ? vec4(cl.r, cl.g, cl.b, 1.0) : vec4(cr.r, cr.g, cr.b, 1.0);
}

[AndrewHazelden]

Stereo Over/Under Output Example

Here is a patched version of the "anaglyph.fp" file that supports Over/Under aka Top/Bottom stereo3D output:

/*
 * Copyright (C) 2020, Inria
 * GRAPHDECO research group, https://team.inria.fr/graphdeco
 * All rights reserved.
 *
 * This software is free for non-commercial, research and evaluation use
 * under the terms of the LICENSE.md file.
 *
 * For inquiries contact sibr@inria.fr and/or George.Drettakis@inria.fr
 */

#version 420

layout(binding = 0) uniform sampler2D left;
layout(binding = 1) uniform sampler2D right;
layout(location= 0) out vec4 out_color;

in vec2 vertex_coord;

void main(void) {
    vec2 texcoord = (vertex_coord + vec2(1.0)) / 2.0;
    vec2 tl = vec2(texcoord.x, texcoord.y / 0.5);
    vec2 tr = vec2(texcoord.x, texcoord.y / 0.5);
    vec4 cl = texture(left, tl);
    vec4 cr = texture(right, tr);
    out_color = vertex_coord.y > 0 ? vec4(cl.r, cl.g, cl.b, 1.0) : vec4(cr.r, cr.g, cr.b, 1.0);
}

[henrypearce4D]

This SBS patch produces an error for me;

[SIBR] --  INFOS  --:   Initializing Raycaster
[SIBR] --  INFOS  --:   Interactive camera using (0.009,1100) near/far planes.
[SIBR] !! WARNING !!:   FILE C:\projects\gauss2\SIBR_viewers\src\core\graphics\Shader.cpp
                        LINE 47, FUNC sibr::GLShader::compileShader
                        GLSL fragment shader compilation failed for program StereoAnaglyph
0(1) : error C0000: syntax error, unexpected '*' at token "*"
0(8) : error C0502: syntax error at token "@"
0(8) : error C0502: syntax error at token "@"
0(12) : error C0204: version directive must be first statement and may not be repeated
0(24) : warning C1503: undefined variable "left"
0(25) : warning C1503: undefined variable "right"
0(26) : warning C1503: undefined variable "out_color"
0(26) : warning C7011: implicit cast from "int" to "float"

Switched to trackball mode.
[SIBR] ##  ERROR  ##:   FILE C:\projects\gauss2\SIBR_viewers\src\core\graphics\Shader.inl
                        LINE 18, FUNC sibr::GLShader::begin
                        OpenGL error 0x0502 (1282) GL_INVALID_OPERATION at C:\projects\gauss2\SIBR_viewers\src\core\graphics\Shader.inl:18

The under over works

[AndrewHazelden]

Hi @henrypearce4D. The leading slash on the first line was not shown in the comment section at the top of the code on line 1. I revised the markdown formatted code snippet block.

[henrypearce4D]

@AndrewHazelden thanks! do you know of a way to fullscreen the pointview window?

[henrypearce4D]

It works quite well, thanks for creating this! Would be nice to be able to full screen the point view window, native VR support will be awesome!

https://github.com/graphdeco-inria/gaussian-splatting/assets/106480554/f57e951e-4680-4ac3-9e75-54bed0ed777d

[Snosixtyboo]

@henrypearce4D the stereo seems to work well, can you make a new issue for the borderless fullscreen? We might be able to address it.

[AndrewHazelden]

Stereo Interlaced Output Example

Here is a patched version of the "anaglyph.fp" file that supports interlaced stereo3D output.

Note: The starting scanline left eye vs right eye view selection can be modified by changing the final line of code from "y % 2 == 0" to "y % 2 == 1".

/*
 * Copyright (C) 2020, Inria
 * GRAPHDECO research group, https://team.inria.fr/graphdeco
 * All rights reserved.
 *
 * This software is free for non-commercial, research and evaluation use
 * under the terms of the LICENSE.md file.
 *
 * For inquiries contact sibr@inria.fr and/or George.Drettakis@inria.fr
 */

#version 420

layout(binding = 0) uniform sampler2D left;
layout(binding = 1) uniform sampler2D right;
layout(location= 0) out vec4 out_color;

in vec2 vertex_coord;

void main(void) {
    vec2 texcoord = (vertex_coord + vec2(1.0)) / 2.0;
    vec4 cl = texture(left, texcoord);
    vec4 cr = texture(right, texcoord);

    int height = textureSize(left, 0).y;
    int y = int(texcoord.y * height);

    // Generate an interlace stereo 3D output by comparing if the scanline row is even or odd using the modulus operator:
    out_color = y % 2 == 0 ? vec4(cl.r, cl.g, cl.b, 1.0) : vec4(cr.r, cr.g, cr.b, 1.0);
}

[cchance27]

this will be interesting when apple releases the vr/ar headset

[nickle-fang]

Can gaussian splatting algorithm directly render equirectangular VR videos like nerfstudio? How to do it?