VirtualGL / virtualgl

Main VirtualGL repository
https://VirtualGL.org
Other
699 stars 106 forks source link

X11 connection rejected because of wrong authentication. #223

Closed Marietto2008 closed 1 year ago

Marietto2008 commented 1 year ago

Hello.

Im trying to setup VirtualGL and TurboVNC on Ubuntu 22.10 and I've found this tutorial as a guide :

https://gist.github.com/cyberang3l/422a77a47bdc15a0824d5cca47e64ba2

From the host (FreeBSD),I connect to the Ubuntu 22.10 VM through ssh issuing the following commands proposed by the tutorial :

root@marietto:/home/marietto # ssh -Y marietto@192.168.1.3
marietto@192.168.1.3's password: 
Welcome to Ubuntu 22.10 (GNU/Linux 5.19.0-28-generic x86_64)

and I've started issuing the commands explained on the cyberang tutorial :

service lightdm stop

root@marietto-BHYVE:/opt/VirtualGL/bin# /opt/VirtualGL/bin/vglserver_config

1) Configure server for use with VirtualGL (GLX + EGL back ends)
2) Unconfigure server for use with VirtualGL (GLX + EGL back ends)
3) Configure server for use with VirtualGL (EGL back end only)
4) Unconfigure server for use with VirtualGL (EGL back end only)
X) Exit

Choose:
1

Restrict 3D X server access to vglusers group (recommended)?
[Y/n]
Y

Restrict framebuffer device access to vglusers group (recommended)?
[Y/n]
Y

Disable XTEST extension (recommended)?
[Y/n]
Y
... Creating vglusers group ...
groupadd: group 'vglusers' already exists
Could not add vglusers group (probably because it already exists.)
... Creating /etc/opt/VirtualGL/ ...
... Granting read permission to /etc/opt/VirtualGL/ for vglusers group ...
... Creating /etc/modprobe.d/virtualgl.conf to set requested permissions for
    /dev/nvidia* ...
... Attempting to remove nvidia module from memory so device permissions
    will be reloaded ...
modprobe: FATAL: Module nvidia_drm is in use.
... Granting write permission to /dev/nvidia-modeset /dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia0 /dev/nvidiactl for vglusers group ...
... Granting write permission to /dev/dri/card0 for vglusers group ...
... Granting write permission to /dev/dri/renderD128 for vglusers group ...
... Modifying /etc/X11/xorg.conf.d/99-virtualgl-dri.conf to enable DRI
    permissions for vglusers group ...
... Modifying /etc/X11/xorg.conf to enable DRI
    permissions for vglusers group ...
... Adding vglgenkey to /etc/gdm3/Init/Default script ...
... Adding vglgenkey to /usr/share/sddm/scripts/Xsetup script ...
... Adding greeter-setup-script=vglgenkey to /etc/lightdm/lightdm.conf ...
... Creating /usr/share/gdm/greeter/autostart/virtualgl.desktop 

Done. You must restart the display manager for the changes to take effect.

REBOOT

root@marietto-BHYVE:/home/marietto# ps ax | grep virtualgl 
nothing.

root@marietto-BHYVE:/home/marietto# xauth merge /opt/VirtualGL/vgl_xauth_key xauth: (argv):1:
merge:  unable to open file /opt/VirtualGL/vgl_xauth_key

root@marietto-BHYVE:/home/marietto# find / -name vgl_xauth_key
nothing.

root@marietto-BHYVE:/home/marietto# xdpyinfo -display :0

Invalid MIT-MAGIC-COOKIE-1 key
xdpyinfo:  unable to open display ":0".

I'm using lightdm :

sudo systemctl disable gdm3

Synchronizing state of gdm3.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable gdm3

root@marietto-BHYVE:/etc/modprobe.d# sudo apt install lightdm
lightdm is already at the most recent version (1.30.0-0ubuntu7).

root@marietto-BHYVE:/etc/modprobe.d# sudo systemctl enable lightdm

Synchronizing state of lightdm.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable lightdm
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.

Possible reasons for having this kind of units are:

• A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some
  instance name specified.

root@marietto-BHYVE:/etc/modprobe.d# ps ax | grep lightdm

    797 ?        Ssl    0:00 /usr/sbin/lightdm
   1037 tty7     Ssl+   0:02 /usr/lib/xorg/Xorg -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
   1222 ?        Sl     0:00 lightdm --session-child 14 17
   7209 pts/1    S+     0:00 grep --color=auto lightdm

This is how I have configurated the config. file needed to make virtualGL works correctly :


/etc/X11/xorg.conf :

Section "DRI"
Mode 0660
Group "vglusers"
EndSection

Section "Device" 
Identifier "Device0" 
Driver "nvidia" 
BusID "PCI:0:8:0"
VendorName "Nvidia Corporation" 
EndSection

/etc/modprobe.d/virtualgl.conf :

options nvidia NVreg_DeviceFileUID=0 NVreg_DeviceFileGID=1002 NVreg_DeviceFileMode=0660

nvidia-graphics-drivers-kms.conf :

options nvidia-drm modeset=1

/etc/X11/xorg.conf.d/99-virtualgl-dri.conf :

Section "DRI" 
Mode 0660
Group "vglusers" 
EndSection

/usr/share/sddm/scripts/Xsetup :

#!/bin/sh 
Xsetup - run as root before the login dialog appears
/opt/VirtualGL/bin/vglgenkey

if [ -e /sbin/prime-offload ]; then 
   echo running NVIDIA Prime setup /sbin/prime-offload    /sbin/prime-offload 
fi

/etc/lightdm/lightdm.conf :

[SeatDefaults] 
[Seat:seat*]
greeter-setup-script=/opt/VirtualGL/bin/vglgenkey

/usr/share/gdm/greeter/autostart/virtualgl.desktop :

[Desktop Entry]
Type=Application
Exec=/opt/VirtualGL/bin/vglgenkey

I have some doubt.

1) modprobe: FATAL: Module nvidia_drm is in use : is this an error that should be fixed by me in some way ?

2) on ubuntu 22.10 I've installed xfce4,that it works well if I "Unconfigure server for use with VirtualGL (GLX + EGL back ends)",instead,if I press 1 on the vglserver_config,it stops to work well. Infact I have some troubles by moving and closing windows,I see an "X" on the monitor,instead of the normal cursor,I can't create files and folders on the desktop. And I don't see any virtualgl process when I do a "ps ax | grep virtualgl" ; it is not launched. Further errors are the following :

root@marietto-BHYVE:/home/marietto# xauth merge /opt/VirtualGL/vgl_xauth_key xauth: (argv):1:  
merge:  unable to open file /opt/VirtualGL/vgl_xauth_key  

root@marietto-BHYVE:/home/marietto# find / -name vgl_xauth_key
nothing. 

root@marietto-BHYVE:/home/marietto# xdpyinfo -display :0
Invalid MIT-MAGIC-COOKIE-1 key xdpyinfo:  unable to open display ":0".

and mostly,the final error when I try to run the virtualGL server :

root@marietto-BHYVE:/opt/VirtualGL/bin/# ./vglgenkey
xauth : file /etc/opt/VirtualGL/vgl_xauth_key does not exist

or :

root@marietto-BHYVE:/etc/modprobe.d# sudo rm /etc/opt/VirtualGL/vgl_xauth_key

root@marietto-BHYVE:/etc/modprobe.d# rm ~/.Xauthority

root@marietto-BHYVE:/etc/modprobe.d# sudo systemctl stop gdm

root@marietto-BHYVE:/etc/modprobe.d# sudo systemctl stop lightdm

root@marietto-BHYVE:/etc/modprobe.d# sudo dpkg-reconfigure gdm

dpkg-query: il pacchetto "gdm" non è installato e non è disponibile alcuna informazione
Usare "dpkg --info" (= "dpkg-deb --info") per esaminare gli archivi.

/usr/sbin/dpkg-reconfigure: gdm is not installed

root@marietto-BHYVE:/etc/modprobe.d# sudo dpkg-reconfigure lightdm

root@marietto-BHYVE:/etc/modprobe.d# sudo /opt/VirtualGL/bin/vglserver_config

1) Configure server for use with VirtualGL (GLX + EGL back ends)
2) Unconfigure server for use with VirtualGL (GLX + EGL back ends)
3) Configure server for use with VirtualGL (EGL back end only)
4) Unconfigure server for use with VirtualGL (EGL back end only)
X) Exit

Choose:
1

Restrict 3D X server access to vglusers group (recommended)?
[Y/n]
Y

Restrict framebuffer device access to vglusers group (recommended)?
[Y/n]
Y

Disable XTEST extension (recommended)?
[Y/n]
Y
... Creating vglusers group ...
groupadd: group 'vglusers' already exists
Could not add vglusers group (probably because it already exists.)
... Granting read permission to /etc/opt/VirtualGL/ for vglusers group ...
... Creating /etc/modprobe.d/virtualgl.conf to set requested permissions for
    /dev/nvidia* ...
... Attempting to remove nvidia module from memory so device permissions
    will be reloaded ...
... Granting write permission to /dev/nvidia-modeset /dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia0 /dev/nvidiactl for vglusers group ...
... Granting write permission to /dev/dri/card0 for vglusers group ...
... Granting write permission to /dev/dri/renderD128 for vglusers group ...
... Modifying /etc/X11/xorg.conf.d/99-virtualgl-dri.conf to enable DRI
    permissions for vglusers group ...
... Modifying /etc/X11/xorg.conf to enable DRI
    permissions for vglusers group ...
... Adding vglgenkey to /etc/gdm3/Init/Default script ...
... Adding vglgenkey to /usr/share/sddm/scripts/Xsetup script ...
... Adding greeter-setup-script=vglgenkey to /etc/lightdm/lightdm.conf ...
... Creating /usr/share/gdm/greeter/autostart/virtualgl.desktop ...

Done. You must restart the display manager for the changes to take effect.

1) Configure server for use with VirtualGL (GLX + EGL back ends)
2) Unconfigure server for use with VirtualGL (GLX + EGL back ends)
3) Configure server for use with VirtualGL (EGL back end only)
4) Unconfigure server for use with VirtualGL (EGL back end only)
X) Exit

Choose:
x

# /opt/VirtualGL/bin/vglgenkey >vglgenkey.out 2>&1
X11 connection rejected because of wrong authentication.
dcommander commented 1 year ago

At first glance, the issue with GDM seems to be related to the fact that Wayland needs to be disabled in GDM. Otherwise, GDM will never run an X server, so there will be no display manager X server for VGL to use as a 3D X server. Normally vglserver_config will detect whether you are using a recent version of GDM that needs to have WaylandEnable=false set in custom.conf. It detects that by looking for ExecStart=/usr/*/gdm or ExecStart=/usr/*/gdm3 in /etc/systemd/system/display-manager.service. Apparently that detection failed on your system for some reason, because vglserver_config never asked you whether you wanted to disable Wayland. That shouldn't have affected LightDM, however, so I'm not sure why you couldn't get LightDM to work.

How this is supposed to work:

I haven't noticed any problems with VirtualGL or TurboVNC under Ubuntu 22.04, so it seems that something may be different about your system.

Marietto2008 commented 1 year ago

I'm using VirtualGL 3.0.90. I've disabled GDM and I have enabled lightdm. Anyway I see this :

/etc/gdm3/custom.conf

#WaylandEnable=false

but when I give a ps ax | grep Xorg it says that Xorg is running. At this point I can try to set WaylandEnable=false anyway on /etc/gdm3/custom.conf. Maybe it works.

On the file /etc/group I have :

vglusers:x:1002:

I'm not experienced,but I suspect that it is wrong. It should be :

vglusers:x:1002:marietto

right ?

dcommander commented 1 year ago

I'm not experienced,but I suspect that it is wrong. It should be :

vglusers:x:1002:marietto

right ?

Yes. That is the likely source of your problems. Add your user account to vglusers, then log out and back in. You should now be able to ls /etc/opt/VirtualGL and see the vgl_xauth_key file there, and

xauth merge /etc/opt/VirtualGL/vgl_xauth_key
xdpyinfo -display :0

should work.

Marietto2008 commented 1 year ago

So,ok,I made those two modifications. Now the xfce problems that I've talked about before are gone. I can use xfce locally (sitting down on the chair ahead the monitor where Ubuntu runs) without problems. But on the folder /opt/VirtualGL/bin/ I don't see any vgl_xauth_key file. I've looked for it everywhere on the disk. It is not generated. Does this means that it is not working ? I have two VirtualGL directories.

1) on /etc/opt/VirtualGL : it is empty

2) on /opt/VirtualGL : here there are all the VirtualGL files,except for the vgl_xauth_key file

dcommander commented 1 year ago

It would be under /etc/opt/VirtualGL, not under /opt/VirtualGL. If it is not under /etc/opt/VirtualGL, then that means that the display manager is not successfully running vglgenkey. I would suggest trying GDM with WaylandEnable=false. I haven't tested LightDM under Ubuntu 22.04 and cannot guarantee that it works, but VirtualGL does officially support GDM under Ubuntu 22.04.

Marietto2008 commented 1 year ago

When on my previous post I've talked about "two modifications" I meant that the first one was to try "WaylandEnable=false". Now it is set like this. Do you want that I comment that line ? Anyway,when I have installed the deb package,it has installed the files under /opt/VirtualGL.

dcommander commented 1 year ago

No, leave it uncommented.

Marietto2008 commented 1 year ago

ok. I kept uncommented WaylandEnable=false and I've reconfigured gdm3 with the following command : dpkg-reconfigure gdm3,choosing gdm3 as default. I don't see any vgl_xauth_key file on the system. And mostly,xfce4 does not start at all. It crashes,giving the error "Oh no! Something has gone wrong. A problem has occurred and the system can't recover".

dcommander commented 1 year ago

I'm sorry. I have no idea. Xfce shouldn't need Wayland, so I don't know why it fails to start with WaylandEnable=false. At this point, my only suggestions are:

dcommander commented 1 year ago

Further comments:

Maybe that will provide a clue as to what is going wrong. Otherwise, I don't have enough information to diagnose the problem.

Marietto2008 commented 1 year ago

I've rebooted the VM and the previous error gone away. Wait a moment : should I install turboVNC server within the Ubuntu / remote VM ? Tutorial says to do that,but at the moment I didn't. Even because my next question would have been "which VNC client do you suggest to me to use on the local system ? "every VNC client can be used with the TurboVNC server ?" "Can I use another VNC server,that has a compatible client on FreeBSD ? I ask this because My local OS is FreeBSD and I see that from the packages I don't have a TurboVNC client. I can choose only between the following ones :

# pkg search vnc

The provides database is up-to-date.

gtk-vnc-1.3.1                  VNC viewer widget for GTK+
neatvnc-0.5.4                  Liberally licensed VNC server library
novnc-1.3.0                    HTML5 VNC client
p5-Net-VNC-0.40_3              Simple VNC client
spiritvnc-0.4.6                Multi-view FLTK-based VNC client
tigervnc-viewer-1.12.0         TigerVNC Viewer
tightvnc-1.3.10_6              Enhanced version of VNC
dcommander commented 1 year ago

Yes, you should install the TurboVNC Server in the VM.

Refer to the User's Guide: https://rawcdn.githack.com/TurboVNC/turbovnc/3.0.2/doc/index.html#hd0010 for compatibility information.

Your best bet among the existing ports is the TigerVNC Viewer, but in order to use the full range of TurboVNC features (including the TurboVNC Session Manager), you need to use the TurboVNC Viewer. It is easy to build the TurboVNC Viewer from source code on FreeBSD. I do it all the time.

I thought that there was already a TurboVNC port for FreeBSD. Someone who was working on a port posted several GitHub issues a while back, so I'm not sure why the port never landed.

Marietto2008 commented 1 year ago

I've been able to connect to the turbovnc server using the tigervnc client 4 FreeBSD (for now) and this is what I see :

Screenshot_2023-01-10_14-29-08

is this GNOME ? it seems you are right. It is using GNOME. But when I login in Ubuntu using my monitor,is not GNOME which starts,but XFCE. I imagine that I should tell to TurboVNC to start XFCE instead of GNOME if I want to use VirtualGL ? How can I do that ?

dcommander commented 1 year ago

I literally just told you how to do that in https://github.com/VirtualGL/virtualgl/issues/223#issuecomment-1377176153.

Marietto2008 commented 1 year ago

I don't have the little gear icon after I enter my username. I have detached the HDMI cable that connect my GPU nvidia 2080 ti to the monitor and I've launched the VM. And then,using the SSH connection to the VM,I have launched the vncserver :

root@marietto-BHYVE:/opt/TurboVNC/bin# /opt/TurboVNC/bin/vncserver

Desktop 'TurboVNC: marietto-BHYVE:1 (marietto)' started on display marietto-BHYVE:1

Starting applications specified in /opt/TurboVNC/bin/xstartup.turbovnc
Log file is /root/.vnc/marietto-BHYVE:1.log

root@marietto-BHYVE:/opt/TurboVNC/bin# vncserver -list

TurboVNC sessions:

X DISPLAY #     PROCESS ID      NOVNC PROCESS ID
:1              5747

Now I can connect using tigervnc client on the address : 192.168.1.3:1,but I still see GNOME as DM,even if I have created the file /root/.vnc/xstartup.turbovnc with the following content inside :

xfce4-session &

as explained here :

https://gist.github.com/cyberang3l/422a77a47bdc15a0824d5cca47e64ba2

where he says :

Make sure that the commands xfce4-panel and xfce4-session exist in your system (if not install them), and append the following line at the end of the file ~/.vnc/xstartup.turbovnc.

xfce4-session &

and I still don't see any vgl_xauth_key file somewhere.

I've created the /root/.vnc/xstartup.turbovnc file from scratch (it didn't exist before). But I see that there is another file called the same on /opt/TurboVNC/bin/xstartup.turbovnc

maybe this is the file that I should modify ? It seems complicated to modify. This is the content :

https://pastebin.ubuntu.com/p/qmnPYfd8FK/

can you explain how can I modify it to start xfce instead of gnome ? very thanks.

dcommander commented 1 year ago

I don't have the little gear icon after I enter my username. I have detached the HDMI cable that connect my GPU nvidia 2080 ti to the monitor and I've launched the VM. And then,using the SSH connection to the VM,I have launched the vncserver :

The gear icon is in the lower right corner.

Screen Shot 2023-01-10 at 9 10 35 AM

If you aren't seeing it, then I have no idea why.

Please understand that VirtualGL and TurboVNC complement each other but are also independent of each other. TurboVNC is completely isolated from the GPU and the physical display, so nothing you do with the physical display will affect it. VirtualGL marshals OpenGL commands from 3D applications to the GPU (via the 3D X server, which the display manager controls) so that those commands can be rendered with GPU acceleration, even in a remote desktop environment (such as TurboVNC) that doesn't have GPU-accelerated OpenGL.

All of that information is in the VirtualGL/TurboVNC User's Guides, along with helpful diagrams.

root@marietto-BHYVE:/opt/TurboVNC/bin# /opt/TurboVNC/bin/vncserver

Desktop 'TurboVNC: marietto-BHYVE:1 (marietto)' started on display marietto-BHYVE:1

Starting applications specified in /opt/TurboVNC/bin/xstartup.turbovnc
Log file is /root/.vnc/marietto-BHYVE:1.log

root@marietto-BHYVE:/opt/TurboVNC/bin# vncserver -list

TurboVNC sessions:

X DISPLAY #     PROCESS ID      NOVNC PROCESS ID
:1              5747

Now I can connect using tigervnc client on the address : 192.168.1.3:1,but I still see GNOME as DM,even if I have created the file /root/.vnc/xstartup.turbovnc with the following content inside :

xfce4-session &

as explained here :

https://gist.github.com/cyberang3l/422a77a47bdc15a0824d5cca47e64ba2

and I still don't see any vgl_xauth_key file somewhere.

I've created the /root/.vnc/xstartup.turbovnc file from scratch (it didn't exist before). But I see that there is another file called the same on /opt/TurboVNC/bin/xstartup.turbovnc

maybe this is the file that I should modify ? It seems complicated to modify. This is the content :

https://pastebin.ubuntu.com/p/qmnPYfd8FK/

can you explain how can I modify it to start xfce instead of gnome ? very thanks.

Let me repeat that I cannot support that tutorial. It is, among other things, outdated. TurboVNC no longer uses ~/.vnc/xstartup.turbovnc. It uses a global xstartup.turbovnc file (/opt/TurboVNC/bin/xstartup.turbovnc) and provides several mechanisms for you to specify a window manager. Those mechanisms, and the appropriate values for specific window managers on specific distributions, are all described at the link I provided earlier: https://turbovnc.org/Documentation/Compatibility30.

Let me also repeat that vgl_xauth_key is generated by vglgenkey, which runs in the context of the display manager. It has nothing to do with TurboVNC. Apparently something is misconfigured on your system, which is preventing GDM from running vglgenkey. I have no idea what is causing that issue. No one else has reported it, and I cannot reproduce it in my own Ubuntu 22.04 VM.

I am extremely busy maintaining three open source projects for a living, and I do not make very much money doing that. I don't have the cycles to provide free personalized support for VirtualGL/TurboVNC users. Please take the time to read the VirtualGL/TurboVNC User's Guides and follow the procedures in those guides rather than in a third-party tutorial. That's the best I can do for you at the moment.

Marietto2008 commented 1 year ago

I can connect to the turboVNC server,but I'm not sure if VirtualGL works correctly. Is there a method to be sure ?

dcommander commented 1 year ago

I can connect to the turboVNC server,but I'm not sure if VirtualGL works correctly. Is there a method to be sure ?

VGL_LOGO=1 vglrun /opt/VirtualGL/bin/glxspheres64

If VirtualGL is working, you should see a "VGL" logo in the corner of the window, and the OpenGL renderer string that it outputs to the command line should say something about your nVidia GPU (as opposed to "Mesa", "llvmpipe", "softpipe", etc., which are all unaccelerated OpenGL implementation.)

I suspect that VGL won't work, because the first thing vglrun will try to do is load /etc/opt/VirtualGL/vgl_xauth_key into your ~/.Xauthority file. That will fail because vgl_xauth_key doesn't exist.

Marietto2008 commented 1 year ago

this works :

VGL_LOGO=1 /opt/VirtualGL/bin/glxspheres64

Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)
GLX FB config ID of window: 0x164 (8/8/8/0)
Visual ID of window: 0x3f7
Context is Direct
OpenGL Renderer: llvmpipe (LLVM 15.0.2, 256 bits)
50.326701 frames/sec - 51.796241 Mpixels/sec

but not this :

VGL_LOGO=1 vglrun /opt/VirtualGL/bin/glxspheres64

Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)
Invalid MIT-MAGIC-COOKIE-1 key
[VGL] ERROR: Could not open display :0.

marietto@marietto-BHYVE:~/Scrivania$ echo $DISPLAY
:1.0

marietto@marietto-BHYVE:~/Scrivania$ vglrun glxgears -display :1.0
Invalid MIT-MAGIC-COOKIE-1 key
[VGL] ERROR: Could not open display :0.

marietto@marietto-BHYVE:~/Scrivania$ vglrun glxgears -display :1
Invalid MIT-MAGIC-COOKIE-1 key
[VGL] ERROR: Could not open display :0.