muskie82 / MonoGS

[CVPR'24 Highlight & Best Demo Award] Gaussian Splatting SLAM
https://rmurai.co.uk/projects/GaussianSplattingSLAM/
Other
1.24k stars 106 forks source link

How to resolve the issue of the visualization interface not displaying? #5

Closed ShilangChen1011 closed 6 months ago

ShilangChen1011 commented 6 months ago

Great job! My system is Ubuntu 20.04, and all the necessary Python packages are installed without any other issues. However, the visualization interface is not showing up. How can I solve this problem?

image

results/tum_datasets_rgbd_dataset_freiburg3_long_office_household/2024-03-06-
10-57-12
FEngine (64 bits) created at 0x381d9f50 (threading is enabled)
FEngine resolved backend: OpenGL
Process Process-4:
Traceback (most recent call last):
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/data/chenshilang_files/MonoGS/gui/slam_gui.py", line 687, in run
    win = SLAM_GUI(params_gui)
  File "/data/chenshilang_files/MonoGS/gui/slam_gui.py", line 66, in __init__
    self.g_renderer = OpenGLRenderer(self.g_camera.w, self.g_camera.h)
  File "/data/chenshilang_files/MonoGS/gui/gl_render/render_ogl.py", line 71, in __init__
    os.path.join(cur_path, "shaders/gau_frag.glsl"),
  File "/data/chenshilang_files/MonoGS/gui/gl_render/util.py", line 90, in load_shaders
    shaders.compileShader(vertex_shader, GL_VERTEX_SHADER),
  File "/home/sirius/anaconda3/envs/MonoGS/lib/python3.7/site-packages/OpenGL/GL/shaders.py", line 226, in compileShader
    shaderType,
RuntimeError: ("Shader compile failure (0): b'0:1(10): error: GLSL 4.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES\\n'", [b'#version 430 core\n\n#define SH_C0 0.28209479177387814f\n#define SH_C1 0.4886025119029199f\n\n#define SH_C2_0 1.0925484305920792f\n#define SH_C2_1 -1.0925484305920792f\n#define SH_C2_2 0.31539156525252005f\n#define SH_C2_3 -1.0925484305920792f\n#define SH_C2_4 0.5462742152960396f\n\n#define SH_C3_0 -0.5900435899266435f\n#define SH_C3_1 2.890611442640554f\n#define SH_C3_2 -0.4570457994644658f\n#define SH_C3_3 0.3731763325901154f\n#define SH_C3_4 -0.4570457994644658f\n#define SH_C3_5 1.445305721320277f\n#define SH_C3_6 -0.5900435899266435f\n\nlayout(location = 0) in vec2 position;\n// layout(location = 1) in vec3 g_pos;\n// layout(location = 2) in vec4 g_rot;\n// layout(location = 3) in vec3 g_scale;\n// layout(location = 4) in vec3 g_dc_color;\n// layout(location = 5) in float g_opacity;\n\n\n#define POS_IDX 0\n#define ROT_IDX 3\n#define SCALE_IDX 7\n#define OPACITY_IDX 10\n#define SH_IDX 11\n\nlayout (std430, binding=0) buffer gaussian_data {\n\tfloat g_data[];\n\t// compact version of following data\n\t// vec3 g_pos[];\n\t// vec4 g_rot[];\n\t// vec3 g_scale[];\n\t// float g_opacity[];\n\t// vec3 g_sh[];\n};\nlayout (std430, binding=1) buffer gaussian_order {\n\tint gi[];\n};\n\nuniform mat4 view_matrix;\nuniform mat4 projection_matrix;\nuniform vec3 hfovxy_focal;\nuniform vec3 cam_pos;\nuniform int sh_dim;\nuniform float scale_modifier;\nuniform int render_mod;  // > 0 render 0-ith SH dim, -1 depth, -2 bill board, -3 gaussian\n\nout vec3 color;\nout float alpha;\nout vec3 conic;\nout vec2 coordxy;  // local coordinate in quad, unit in pixel\n\nmat3 computeCov3D(vec3 scale, vec4 q)  // should be correct\n{\n    mat3 S = mat3(0.f);\n    S[0][0] = scale.x;\n\tS[1][1] = scale.y;\n\tS[2][2] = scale.z;\n\tfloat r = q.x;\n\tfloat x = q.y;\n\tfloat y = q.z;\n\tfloat z = q.w;\n\n    mat3 R = mat3(\n\t\t1.f - 2.f * (y * y + z * z), 2.f * (x * y - r * z), 2.f * (x * z + r * y),\n\t\t2.f * (x * y + r * z), 1.f - 2.f * (x * x + z * z), 2.f * (y * z - r * x),\n\t\t2.f * (x * z - r * y), 2.f * (y * z + r * x), 1.f - 2.f * (x * x + y * y)\n\t);\n\n    mat3 M = S * R;\n    mat3 Sigma = transpose(M) * M;\n    return Sigma;\n}\n\nvec3 computeCov2D(vec4 mean_view, float focal_x, float focal_y, float tan_fovx, float tan_fovy, mat3 cov3D, mat4 viewmatrix)\n{\n    vec4 t = mean_view;\n    // why need this? Try remove this later\n    float limx = 1.3f * tan_fovx;\n    float limy = 1.3f * tan_fovy;\n    float txtz = t.x / t.z;\n    float tytz = t.y / t.z;\n    t.x = min(limx, max(-limx, txtz)) * t.z;\n    t.y = min(limy, max(-limy, tytz)) * t.z;\n\n    mat3 J = mat3(\n        focal_x / t.z, 0.0f, -(focal_x * t.x) / (t.z * t.z),\n\t\t0.0f, focal_y / t.z, -(focal_y * t.y) / (t.z * t.z),\n\t\t0, 0, 0\n    );\n    mat3 W = transpose(mat3(viewmatrix));\n    mat3 T = W * J;\n\n    mat3 cov = transpose(T) * transpose(cov3D) * T;\n    // Apply low-pass filter: every Gaussian should be at least\n\t// one pixel wide/high. Discard 3rd row and column.\n\tcov[0][0] += 0.3f;\n\tcov[1][1] += 0.3f;\n    return vec3(cov[0][0], cov[0][1], cov[1][1]);\n}\n\nvec3 get_vec3(int offset)\n{\n\treturn vec3(g_data[offset], g_data[offset + 1], g_data[offset + 2]);\n}\nvec4 get_vec4(int offset)\n{\n\treturn vec4(g_data[offset], g_data[offset + 1], g_data[offset + 2], g_data[offset + 3]);\n}\n\nvoid main()\n{\n\tint boxid = gi[gl_InstanceID];\n\tint total_dim = 3 + 4 + 3 + 1 + sh_dim;\n\tint start = boxid * total_dim;\n\tvec4 g_pos = vec4(get_vec3(start + POS_IDX), 1.f);\n    vec4 g_pos_view = view_matrix * g_pos;\n    vec4 g_pos_screen = projection_matrix * g_pos_view;\n\tg_pos_screen.xyz = g_pos_screen.xyz / g_pos_screen.w;\n    g_pos_screen.w = 1.f;\n\t// early culling\n\tif (any(greaterThan(abs(g_pos_screen.xyz), vec3(1.3))))\n\t{\n\t\tgl_Position = vec4(-100, -100, -100, 1);\n\t\treturn;\n\t}\n\tvec4 g_rot = get_vec4(start + ROT_IDX);\n\tvec3 g_scale = get_vec3(start + SCALE_IDX);\n\tfloat g_opacity = g_data[start + OPACITY_IDX];\n\n    mat3 cov3d = computeCov3D(g_scale * scale_modifier, g_rot);\n    vec2 wh = 2 * hfovxy_focal.xy * hfovxy_focal.z;\n    vec3 cov2d = computeCov2D(g_pos_view, \n                              hfovxy_focal.z, \n                              hfovxy_focal.z, \n                              hfovxy_focal.x, \n                              hfovxy_focal.y, \n                              cov3d, \n                              view_matrix);\n\n    // Invert covariance (EWA algorithm)\n\tfloat det = (cov2d.x * cov2d.z - cov2d.y * cov2d.y);\n\tif (det == 0.0f)\n\t\tgl_Position = vec4(0.f, 0.f, 0.f, 0.f);\n    \n    float det_inv = 1.f / det;\n\tconic = vec3(cov2d.z * det_inv, -cov2d.y * det_inv, cov2d.x * det_inv);\n    \n    vec2 quadwh_scr = vec2(3.f * sqrt(cov2d.x), 3.f * sqrt(cov2d.z));  // screen space half quad height and width\n    vec2 quadwh_ndc = quadwh_scr / wh * 2;  // in ndc space\n    g_pos_screen.xy = g_pos_screen.xy + position * quadwh_ndc;\n    coordxy = position * quadwh_scr;\n    gl_Position = g_pos_screen;\n    \n    alpha = g_opacity;\n\n\tif (render_mod == -1)\n\t{\n\t\tfloat depth = -g_pos_view.z;\n\t\tdepth = depth < 0.05 ? 1 : depth;\n\t\tdepth = 1 / depth;\n\t\tcolor = vec3(depth, depth, depth);\n\t\treturn;\n\t}\n\n\t// Covert SH to color\n\tint sh_start = start + SH_IDX;\n\tvec3 dir = g_pos.xyz - cam_pos;\n    dir = normalize(dir);\n\tcolor = SH_C0 * get_vec3(sh_start);\n\t\n\tif (sh_dim > 3 && render_mod >= 1)  // 1 * 3\n\t{\n\t\tfloat x = dir.x;\n\t\tfloat y = dir.y;\n\t\tfloat z = dir.z;\n\t\tcolor = color - SH_C1 * y * get_vec3(sh_start + 1 * 3) + SH_C1 * z * get_vec3(sh_start + 2 * 3) - SH_C1 * x * get_vec3(sh_start + 3 * 3);\n\n\t\tif (sh_dim > 12 && render_mod >= 2)  // (1 + 3) * 3\n\t\t{\n\t\t\tfloat xx = x * x, yy = y * y, zz = z * z;\n\t\t\tfloat xy = x * y, yz = y * z, xz = x * z;\n\t\t\tcolor = color +\n\t\t\t\tSH_C2_0 * xy * get_vec3(sh_start + 4 * 3) +\n\t\t\t\tSH_C2_1 * yz * get_vec3(sh_start + 5 * 3) +\n\t\t\t\tSH_C2_2 * (2.0f * zz - xx - yy) * get_vec3(sh_start + 6 * 3) +\n\t\t\t\tSH_C2_3 * xz * get_vec3(sh_start + 7 * 3) +\n\t\t\t\tSH_C2_4 * (xx - yy) * get_vec3(sh_start + 8 * 3);\n\n\t\t\tif (sh_dim > 27 && render_mod >= 3)  // (1 + 3 + 5) * 3\n\t\t\t{\n\t\t\t\tcolor = color +\n\t\t\t\t\tSH_C3_0 * y * (3.0f * xx - yy) * get_vec3(sh_start + 9 * 3) +\n\t\t\t\t\tSH_C3_1 * xy * z * get_vec3(sh_start + 10 * 3) +\n\t\t\t\t\tSH_C3_2 * y * (4.0f * zz - xx - yy) * get_vec3(sh_start + 11 * 3) +\n\t\t\t\t\tSH_C3_3 * z * (2.0f * zz - 3.0f * xx - 3.0f * yy) * get_vec3(sh_start + 12 * 3) +\n\t\t\t\t\tSH_C3_4 * x * (4.0f * zz - xx - yy) * get_vec3(sh_start + 13 * 3) +\n\t\t\t\t\tSH_C3_5 * z * (xx - yy) * get_vec3(sh_start + 14 * 3) +\n\t\t\t\t\tSH_C3_6 * x * (xx - 3.0f * yy) * get_vec3(sh_start + 15 * 3);\n\t\t\t}\n\t\t}\n\t}\n\tcolor += 0.5f;\n}\n'], GL_VERTEX_SHADER)
koktavy commented 6 months ago

I've run into a similar issue on WSL2 Ubuntu 22, with GLSL support up to 4.20. I'm in the process of trying to downgrade the shader but if an earlier solution is available I'm all ears!

Thanks for your work.

rmurai0610 commented 6 months ago

Hi @earc-chen, it looks like your opengl version is too low. You'll need at least GLSL 4.3 to run the ellipsoid shader. We've not encountered this issue but these seem to be related: https://github.com/mp3guy/ElasticFusion/issues/188 https://github.com/mp3guy/ElasticFusion/issues/96 It looks like that if you have Nvidia GPU and intel GPU, you need to make sure that Nvidia GPU is selected

rmurai0610 commented 6 months ago

Hi @koktavy for WSL2, we don't have the setup to replicate, but this might be useful: https://github.com/microsoft/WSL/issues/9753#issuecomment-1459626307 Hope this helps!

koktavy commented 6 months ago

I hadn't seen that solution. It doesn't seem to have an effect in this case.

I've built a custom kernel to enable camera support, so that may have broken the connection to the GPU (NVIDIA 3090).

$LIBGL_ALWAYS_SOFTWARE=true glxinfo -B

name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 15.0.7, 256 bits) (0xffffffff)
    Version: 23.2.1
    Accelerated: no
    Video memory: 32056MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
    VBO free memory - total: 31 MB, largest block: 31 MB
    VBO free aux. memory - total: 24582 MB, largest block: 24582 MB
    Texture free memory - total: 31 MB, largest block: 31 MB
    Texture free aux. memory - total: 24582 MB, largest block: 24582 MB
    Renderbuffer free memory - total: 31 MB, largest block: 31 MB
    Renderbuffer free aux. memory - total: 24582 MB, largest block: 24582 MB
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 1194463 MB
    Total available memory: 1226520 MB
    Currently available dedicated video memory: 31 MB
OpenGL vendor string: Mesa
OpenGL renderer string: llvmpipe (LLVM 15.0.7, 256 bits)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 4.5 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL shading language version string: 4.50
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile

OpenGL ES profile version string: OpenGL ES 3.2 Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
$ python slam.py --config configs/mono/tum/fr3_office.yaml

WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
MonoGS: saving results in results/datasets_tum/2024-03-06-15-31-26
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
FEngine (64 bits) created at 0x545dfd00 (threading is enabled)
FEngine resolved backend: OpenGL
Process Process-4:
Traceback (most recent call last):
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/tavius/MonoGS/gui/slam_gui.py", line 687, in run
    win = SLAM_GUI(params_gui)
  File "/home/tavius/MonoGS/gui/slam_gui.py", line 66, in __init__
    self.g_renderer = OpenGLRenderer(self.g_camera.w, self.g_camera.h)
  File "/home/tavius/MonoGS/gui/gl_render/render_ogl.py", line 71, in __init__
    os.path.join(cur_path, "shaders/gau_frag.glsl"),
  File "/home/tavius/MonoGS/gui/gl_render/util.py", line 90, in load_shaders
    shaders.compileShader(vertex_shader, GL_VERTEX_SHADER),
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/site-packages/OpenGL/GL/shaders.py", line 241, in compileShader
    shaderType,
OpenGL.GL.shaders.ShaderCompilationError: ("Shader compile failure (0): b'0:1(10): error: GLSL 4.30 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.50, 3.30, 4.00, 4.10, 4.20, 1.00 ES, 3.00 ES, and 3.10 ES\\n'", [b'#version 430 core\n\n#define SH_C0 0.28209479177387814f\n#define SH_C1 0.4886025119029199f\n\n#define SH_C2_0 1.0925484305920792f\n#define SH_C2_1 -1.0925484305920792f\n#define
...

Though I can run nvidia-smi with a detected GPU

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.60.01              Driver Version: 551.76         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:09:00.0  On |                  N/A |
|  0%   43C    P8             47W /  390W |    1971MiB /  24576MiB |      8%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        22      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+
rmurai0610 commented 6 months ago

I see, I'm not too familiar with WSL so I'm not sure if I can help much here. Have you tried running with --eval so it runs without GUI? (just to confirm that it's only the glsl versions problem, not Cuda etc)

koktavy commented 6 months ago

Seems there's more going wrong than just GLSL for me. It's ok if there's not an easy solve here, I understand that I'm working outside of the expected configuration for this project.

$ python slam.py --config configs/mono/tum/fr3_office.yaml --eval

WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
MonoGS: Running MonoGS in Evaluation Mode
MonoGS: Following config will be overriden
MonoGS:         save_results=True
MonoGS:         use_gui=False
MonoGS:         eval_rendering=True
MonoGS:         use_wandb=True
MonoGS: saving results in results/datasets_tum/2024-03-06-17-37-58
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.16.3
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
Traceback (most recent call last):
  File "slam.py", line 252, in <module>
    slam = SLAM(config, save_dir=save_dir)
  File "slam.py", line 110, in __init__
    self.frontend.run()
  File "/home/tavius/MonoGS/utils/slam_frontend.py", line 361, in run
    monocular=self.monocular,
  File "/home/tavius/MonoGS/utils/eval_utils.py", line 70, in eval_ate
    latest_frame_idx = kf_ids[-1] + 2 if final else kf_ids[-1] + 1
IndexError: list index out of range
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/tavius/MonoGS/wandb/offline-run-20240306_173801-zuq7iwap
wandb: Find logs at: ./wandb/offline-run-20240306_173801-zuq7iwap/logs
WARNING: CPU random generator seem to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/tavius/miniconda3/envs/MonoGS/lib/python3.7/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
ShilangChen1011 commented 6 months ago

The successful operation can be achieved by adding the following environment variable settings in the console. export MESA_GL_VERSION_OVERRIDE=4.3

rmurai0610 commented 6 months ago

@koktavy Sorry, I'm not familiar with wsl so I don't think we can help here.

WARNING: CPU random generator seems to be failing, disabling hardware random number generation
WARNING: RDRND generated: 0xffffffff 0xffffffff 0xffffffff 0xffffffff

is not from MonoGS but from your machine and with a quick search, it seems to be something to do with your BIOS? https://stackoverflow.com/questions/66290789/how-to-solve-cpu-random-generator-seem-to-be-failing-disabling-hardware-random