GLMakie on headless system through EGL?

[aramirezreyes]

Hi!

I would like to get some help here. I am trying to run GLMakie on the Perlmutter supercomputer. I want to profit from its GPUs to do some nice visualizations using GLMakie.

Two possibilities: One is on Jupyter through jupyterhub and one through ssh.

The catch: there is no X11 server and no xvfb-run. I opened a ticket with support and I got this answer back:

Perlmutter has NVIDIA OpenGL installed. It is in the usual place, /usr/lib64/libOpenGL.so OpenGL based rendering software I've installed on perlmutter has been able to find and use it without issue. One thing to be aware of on a system like Perlmutter is that x11 is not running. As such the rendering software in question must have some extra code to set the headless rendering context up. A couple of ways that this can be done are through EGL, which is a newer addition to OpenGL standard specifically for this purpose, or by using OSMesa, which is an older software only solution. NVIDIA's EGL is installed and available on Perlmutter /usr/lib64/libEGL.so. All this to say, the software you are trying to use both needs to have the EGL or OSMesa capability and it needs to be compiled with those features enabled.

Note that OpenGL based rendering codes that do not have EGL/OSMesa capability could potentially still run on a system like Perlmutter, but you would have to start an X11 server in your batch script. Assuming that x11 is installed, the tricky part about that is getting an "xorg.conf" file that sets x11 up for this headless mode. It is possible to do this way, I've done it on other systems, but to my knowledge no one does it that way anymore thanks to the availability of EGL. Doing this without EGL/OSMesa is a lot of work and requires enough skill with X11 to configure it in headless mode, and enough skill with sys admins to get them to install x11 in the first place, assuming it is not already installed which I have not checked.

From what I can see, GLFW does suppport the thing called EGL, and for python thought the library moderngl the way to use it would be like this

import moderngl
ctx = moderngl.create_context(
    standalone=True,
    backend='egl',
    # These are OPTIONAL if you want to load a specific version
    libgl='libGL.so.1',
    libegl='libEGL.so.1',
)

Is it possible to make GLMakie run with EGL?

Thanks.

[SimonDanisch]

Can you try creating a window with GLFW.jl? https://www.glfw.org/docs/latest/window_guide.html#window_hints_ctx I'm not 100% sure how to put this together, but if you can create a GLFW window with those docs, we should be able to get GLMakie to run... It does look a bit like GLFW should just automatically use EGL if present?

[aramirezreyes]

Thanks, Simon!

So I did the following:

aramreye@perlmutter:login31:~/TestGLMakie> export DISPLAY=:99.0
aramreye@perlmutter:login31:~/TestGLMakie> Xvfb :99 -screen 0 640x480x24 &
[1] 261483

aramreye@perlmutter:login31:~/TestGLMakie> julia --project=@.
aramreye@perlmutter:login31:~/TestGLMakie> julia --project=@.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.0-beta4 (2023-02-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using GLFW

julia> ch = GLFW.standard_context_hints(3,3)
4-element Vector{Tuple{UInt32, Integer}}:
 (0x00022002, 3)
 (0x00022003, 3)
 (0x00022006, 1)
 (0x00022008, 0x00032001)

julia> push!(ch,(GLFW.CONTEXT_CREATION_API,GLFW.EGL_CONTEXT_API))
5-element Vector{Tuple{UInt32, Integer}}:
 (0x00022002, 3)
 (0x00022003, 3)
 (0x00022006, 1)
 (0x00022008, 0x00032001)
 (0x0002200b, 0x00036002)

julia> GLFW.Window(;contexthints=ch)
ERROR: GLFWError (API_UNAVAILABLE): EGL: Failed to get EGL display: Success
Stacktrace:
 [1] _ErrorCallbackWrapper(code::Int32, description::Cstring)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/callback.jl:43
 [2] CreateWindow(width::Int64, height::Int64, title::String, monitor::GLFW.Monitor, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:499
 [3] GLFW.Window(; name::String, resolution::Tuple{Int64, Int64}, debugging::Bool, major::Int64, minor::Int64, windowhints::Vector{Tuple{UInt32, Int64}}, contexthints::Vector{Tuple{UInt32, Integer}}, visible::Bool, focus::Bool, fullscreen::Bool, monitor::Nothing, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:344
 [4] kwcall(::NamedTuple{(:contexthints,), Tuple{Vector{Tuple{UInt32, Integer}}}}, ::Type{GLFW.Window})
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:302
 [5] top-level scope
   @ REPL[7]:1

Should I do something to manually point GLFW.jl to the suggested libraries of the first post?

[aramirezreyes]

Wait, I probably tried the incorrect constructor. Doing

julia> window = GLFW.CreateWindow(640, 480, "GLFW.jl")
ERROR: GLFWError (FORMAT_UNAVAILABLE): EGL: Failed to find a suitable EGLConfig

Seems like it is going in the right direction...

[Yaraslaut]

You can draw directly in repl session via SixelTerm

using CairoMakie
using SixelTerm

One thing to note that you need terminal that suport sixel graphics

[SimonDanisch]

https://github.com/glfw/glfw/blob/d299d9f78857e921b66bdab42c7ea27fe2e31810/src/egl_context.c#L202-L232

You seem to have the wrong window hints. What does GLFW.Window() actually do?

[aramirezreyes]

You can draw directly in repl session via SixelTerm

using CairoMakie
using SixelTerm

One thing to note that you need terminal that suport sixel graphics

Thanks @Yaraslaut! I do use CairoMakie in this system. However, I a exploring the possibility of profiting from the gpus that it has!

[aramirezreyes]

https://github.com/glfw/glfw/blob/d299d9f78857e921b66bdab42c7ea27fe2e31810/src/egl_context.c#L202-L232

You seem to have the wrong window hints. What does GLFW.Window() actually do?

By itself I get this:

julia> GLFW.Window()
ERROR: GLFWError (VERSION_UNAVAILABLE): GLX: Forward compatibility requested but GLX_ARB_create_context_profile is unavailable
Stacktrace:
 [1] _ErrorCallbackWrapper(code::Int32, description::Cstring)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/callback.jl:43
 [2] CreateWindow(width::Int64, height::Int64, title::String, monitor::GLFW.Monitor, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:499
 [3] GLFW.Window(; name::String, resolution::Tuple{Int64, Int64}, debugging::Bool, major::Int64, minor::Int64, windowhints::Vector{Tuple{UInt32, Int64}}, contexthints::Vector{Tuple{UInt32, Integer}}, visible::Bool, focus::Bool, fullscreen::Bool, monitor::Nothing, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:344
 [4] GLFW.Window()
   @ GLFW ~/.julia/packages/GLFW/BWxfF/src/glfw3.jl:302
 [5] top-level scope
   @ REPL[2]:1

[aramirezreyes]

Thanks for the patience, Simon. I know almost nothing about this. If I look here, it appears that this error comes from glx_context.c but there is another file egl_context.c, does it mean that we are not detecting the context correctly?

Now, if I set the context hints again I do get the weird error message of succes:

julia> using GLFW

julia> ch = GLFW.standard_context_hints(3,3)
4-element Vector{Tuple{UInt32, Integer}}:
 (0x00022002, 3)
 (0x00022003, 3)
 (0x00022006, 1)
 (0x00022008, 0x00032001)

julia> GLFW.Window(;contexthints=ch)
ERROR: GLFWError (API_UNAVAILABLE): EGL: Failed to get EGL display: Success
Stacktrace:
 [1] _ErrorCallbackWrapper(code::Int32, description::Cstring)
   @ GLFW ~/.julia/packages/GLFW/OAHhH/src/callback.jl:43
 [2] CreateWindow(width::Int64, height::Int64, title::String, monitor::GLFW.Monitor, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/OAHhH/src/glfw3.jl:504
 [3] GLFW.Window(; name::String, resolution::Tuple{Int64, Int64}, debugging::Bool, major::Int64, minor::Int64, windowhints::Vector{Tuple{UInt32, Int64}}, contexthints::Vector{Tuple{UInt32, Integer}}, visible::Bool, focus::Bool, fullscreen::Bool, monitor::Nothing, share::GLFW.Window)
   @ GLFW ~/.julia/packages/GLFW/OAHhH/src/glfw3.jl:344
 [4] kwcall(::NamedTuple{(:contexthints,), Tuple{Vector{Tuple{UInt32, Integer}}}}, ::Type{GLFW.Window})
   @ GLFW ~/.julia/packages/GLFW/OAHhH/src/glfw3.jl:302
 [5] top-level scope
   @ REPL[5]:1

[SimonDanisch]

Try just setting the egl windowhint before GLFW.Window(),theyre global state

[aramirezreyes]

Ah! Thanks!

I seem to understand that glfw does not support this kind of yet: https://github.com/glfw/glfw/issues/1882

There are two stale PRs:

https://github.com/glfw/glfw/pull/1608 https://github.com/glfw/glfw/issues/1592