search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

How do I run OpenGL app inside container #4290

Closed tashrifbillah closed 5 years ago

tashrifbillah commented 5 years ago

I am trying to run fsleyes, an app requiring OpenGL. I do have a container here. However, for faster reproducibility of the issue, I am showing a basic Centos7 image below:

Version of Singularity:

singularity version 3.2.1-1.1.el7

Expected behavior

Something like:

server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL core profile version string: 4.4.0 NVIDIA 410.78
OpenGL core profile shading language version string: 4.40 NVIDIA via Cg compiler
OpenGL version string: 4.6.0 NVIDIA 410.78
OpenGL shading language version string: 4.60 NVIDIA
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 410.78
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
    GL_EXT_shader_implicit_conversions, GL_EXT_shader_integer_mix,

Actual behavior

libGL: screen 0 does not appear to be DRI2 capable
libGL: OpenDriver: trying /usr/lib64/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib64/dri/swrast_dri.so
libGL: Can't open configuration file /root/.drirc: No such file or directory.
libGL: Can't open configuration file /root/.drirc: No such file or directory.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Error: couldn't find RGB GLX visual or fbconfig

Steps to reproduce behavior

singularity pull centos
singularity shell --contain --writable-tmpfs centos_latest.sif

Inside the container:

yum -y install glx-utils mesa-dri-drivers.x86_64
export LIBGL_DEBUG=verbose
glxinfo | grep version

More information

Singularity fsl_tbss.img:~> echo $DISPLAY
:0

Singularity fsl_tbss.img:~> ls /usr/lib64/dri/
i915_dri.so  kms_swrast_dri.so  r300_dri.so  radeonsi_dri.so  virtio_gpu_dri.so
i965_dri.so  nouveau_dri.so     r600_dri.so  swrast_dri.so    vmwgfx_dri.so
jmstover commented 5 years ago

I'm pulling the image to test... but are you using the --nv flag to bring the host nvidia libraries into the container?

tashrifbillah commented 5 years ago

Should I? Can you give a full pull command? I am not aware of any --nv flag with singularity pull. I wrote everything I did.

jmstover commented 5 years ago

Here's what I did:

$ xhost +
$ sudo singularity shell --nv --writable-tmpfs fsl_tbss_0.0.1.sif
> yum -y install glx-utils which
> ldd `which glxinfo`
> glxinfo | grep 'version'

The xhost + was to allow connections to the X server... you may not need that (And that opens it wide open as well). Adding the which package, was just for my usage on the ldd command.

But, I can run glxinfo and glxgears

What the --nv flag does, is it takes the host libraries (which can be driver dependent), and bind mounts them into the container on run. Then modifies LD_LIBRARY_PATH so the bind location is looked at first.

tashrifbillah commented 5 years ago

Testing ...

tashrifbillah commented 5 years ago

Okay, the above assumes host computer has Nvidia driver. What if the host doesn't have one, would it be possible to run OpenGL apps?

tashrifbillah commented 5 years ago

Also, I need to run the same on a windows computer. I have followed this instruction to install singularity, and below is my Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "sylabs/singularity-3.3-centos-7-64"
  config.ssh.forward_x11 = true
end

Willingly, I am not using config.vm.box = "sylabs/singularity-3.2-ubuntu-bionic64 image.

Now when I do:

vagrant up
vagrant ssh

And inside singularity box:

yum install -y glx-utils
glxgears/glxinfo

I get:

Error: unable to open display

There are a few references, but I have not been successful so far:

https://coderwall.com/p/ozhfva/run-graphical-programs-within-vagrantboxes

https://serverfault.com/questions/422908/how-can-i-prevent-the-warning-no-xauth-data-using-fake-authentication-data-for

jmstover commented 5 years ago

Yes, but you'd then need to have libraries that are compatible with the driver that is used by the X server. In the nvidia case, we're using the nvidia libraries and assuming that the X configuration is correct for the NV graphics card.

If we assume you're using something else, then theoretically the Mesa GL libraries will work okay, but it will depend on how the X server configuration is done.

I need to run the same on a windows computer.

Do you have an X server running on the Windows system? Like, what is it... Xming? Maybe cygwin has a X11 build?

tashrifbillah commented 5 years ago
  1. I have https://sourceforge.net/projects/vcxsrv/ running.
  2. I have made the following edits already:

/etc/ssh/ssh_config

Host *
   ForwardAgent yes
   ForwardX11 yes
   ForwardX11Trusted yes
   Port 22
   Protocol 2
   XAuthLocation "/C/Program Files/VcXsrv/xauth"

/etc/ssh/sshd_config

X11Forwarding yes
  1. With the provided Vagrantfile, I am doing:
vagrant up
vagrant ssh -- -vvv -X

Still, getting following error messages:

Warning: No xauth data; using fake authentication data for X11 forwarding.

Here is the complete log:

OpenSSH_7.7p1, OpenSSL 1.0.2o  27 Mar 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug2: resolve_canonicalize: hostname 127.0.0.1 is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file C:/Users/tashr/vm-singularity/.vagrant/machines/default/virtualbox/private_key type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:/Users/tashr/vm-singularity/.vagrant/machines/default/virtualbox/private_key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 127.0.0.1:2222 as 'vagrant'
debug3: put_host_port: [127.0.0.1]:2222
debug3: hostkeys_foreach: reading file "/dev/null"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: zlib@openssh.com,zlib,none
debug2: compression stoc: zlib@openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: zlib@openssh.com
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64-etm@openssh.com compression: zlib@openssh.com
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:KaEwYqzCYZiRknBrdfdftHqlrKrHC2w6xqXpx92OGAM
debug3: put_host_port: [127.0.0.1]:2222
debug3: put_host_port: [127.0.0.1]:2222
debug3: hostkeys_foreach: reading file "/dev/null"
debug1: checking without port identifier
debug3: hostkeys_foreach: reading file "/dev/null"
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: C:/Users/tashr/vm-singularity/.vagrant/machines/default/virtualbox/private_key (0x0), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:/Users/tashr/vm-singularity/.vagrant/machines/default/virtualbox/private_key
debug3: sign_and_send_pubkey: RSA SHA256:HPFIoB8h3N6lUdER1ddeW4ZQPQnA0chbFcSi7O7ouj0
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 52
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 127.0.0.1 ([127.0.0.1]:2222).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: exec
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: x11_get_proto: /C/Program Files/VcXsrv/xauth  list needs-to-be-defined 2>/dev/null
Warning: No xauth data; using fake authentication data for X11 forwarding.
debug1: Requesting X11 forwarding with authentication spoofing.
debug2: channel 0: request x11-req confirm 1
debug3: send packet: type 98
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 4
debug1: Remote: No xauth program; cannot forward with spoofing.
debug3: receive packet: type 100
debug2: channel_input_status_confirm: type 100 id 0
X11 forwarding request failed on channel 0
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

However, I don't think the above correction created any .Xauthority file in my home folder.

jmstover commented 5 years ago

What does vagrant ssh use to connect? For straight ssh, I'll normally use both options -X -Y (ForwardX11, TrustedX11).

In your container image, also make sure the xorg-x11-xauth package is installed.

tashrifbillah commented 5 years ago

What does vagrant ssh use to connect?

Don't quite understand.

For the suggestion, what is the order, vagrant -X -Y ssh ?

jmstover commented 5 years ago

Normal ssh client, I'll do: ssh -X -Y user@host

I've never used Vagrant before, so I'm guessing here on what it's doing in the background when you run vagrant ssh ...

tashrifbillah commented 5 years ago

I've never used Vagrant before

I guess you work on Linux only since vagrant should be the only gateway between MAC/Windows and Singularity container.

Normal ssh client, I'll do: ssh -X -Y user@host

I understand the syntax for normal ssh. However, in this case, vagrant -X -Y ssh was required. I guess you noticed, I didn't have to specify any user@host. In fact, this is the instruction in the third box here.

In any case, I have not been able to launch glxinfo through Windows singularity box. Though I did ssh like you suggested, echo $DISPLAY is blank inside the singularity container.

jmstover commented 5 years ago

Though I did ssh like you suggested, echo $DISPLAY is blank inside the singularity container.

Do you have xauth in the container? I know it isn't in the image... xauth list should give you a listing of .Xauthority entries. Beyond that, the X server needs to allow remote connections.

work on Linux only

The bulk yes.... so I'm guessing here just based on how the services/protocol should be responding normally.

tashrifbillah commented 5 years ago

Okay, I solved the xauth issue, but now I have the following issue with glxgears. In fact this is a reproduction of larger issue I am having with fsleyes.

Singularity recipe:

Bootstrap: docker
From: centos:7.5.1804

%post
    yum -y install xauth xclock glx-utils mesa-dri-drivers.x86_64

%environment
    export LIBGL_DEBUG=verbose
    export LC_ALL=C

Build:

singularity build xclock-glxgears Singularity

You should be able to pull my image as follows:

singularity pull xclock-glxgears library://tbillah/collection/xclock-glxgears

As before, I have the vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "sylabs/singularity-3.3-centos-7-64"
  config.ssh.forward_x11 = true
end

Then, I do the following on Windows terminal (VcXrv is on) :

export DISPLAY=localhost:0
vagrant up
vagrant ssh

. (inside box)

cd /tmp
singularity pull xclock-glxgears library://tbillah/collection/xclock-glxgears
singularity shell --contain --writable-tmpfs xclock-glxgears

(inside container)

xeyes       # runs successfully
xclock      # runs successfully
glxgears   # runs into following error

Error

libGL: screen 0 does not appear to be DRI2 capable
libGL: OpenDriver: trying /usr/lib64/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib64/dri/swrast_dri.so
libGL: Can't open configuration file /root/.drirc: No such file or directory.
libGL: Can't open configuration file /root/.drirc: No such file or directory.
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Error: couldn't get an RGB, Double-buffered visual
dtrudg commented 5 years ago

Hi @tashrifbillah - I suspect that mesa-dri is not what you want here. dri is about rendering with direct access to hardware. Your VM likely doesn't have virtual hardware useful for that - do you get anything different if you just use mesa-libGLU and libGL?

Also you can try setting export LIBGL_ALWAYS_INDIRECT=1

I'm afraid I don't have a Windows machine available at present, but I recall setting INDIRECT being important for VcXsrv a few years back.

tashrifbillah commented 5 years ago

I am still working on it. Will post my thoughts soon.

tashrifbillah commented 5 years ago

Hi, thanks for your support so far.

Here is my organized repository. Hope it will be helpful to address this issue in future. Thanks.

https://github.com/tashrifbillah/glxgears-containers