Service starts too early to find resolutionKMS in Debian Testing

meeas commented 6 years ago

I'm running Debian Testing and just updated my install and found open-vm-tools failed to resize resolution in my VM after reboot. If I started the service with systemctl by hand after I had logged in, everything worked fine. Digging in the the logs, I found that when the service started on boot, it would throw this error in /var/log/vmware-vmsvc.log:

[Dec 09 23:18:53.585] [ warning] [resolutionCommon] resolutionCheckForKMS: No system support for resolutionKMS.

However when starting it by hand, /var/log/vmware-vmsvc.log would show:

[Dec 09 23:28:49.781] [ message] [resolutionCommon] resolutionCheckForKMS: System support available for resolutionKMS. [Dec 09 23:28:49.781] [ message] [vmtoolsd] Plugin 'resolutionKMS' initialized.

I'm guessing that open-vm-tools is starting too early so I added a dependency to start after gdm.service which worked like a charm. Not sure if this is the right permanent solution, but thought I'd pass it along.

$ cat /etc/systemd/system/multi-user.target.wants/open-vm-tools.service [Unit] Description=Service for virtual machines hosted on VMware Documentation=http://open-vm-tools.sourceforge.net/about.php ConditionVirtualization=vmware DefaultDependencies=no After=gdm.service Before=cloud-init-local.service

[Service] ExecStart=/usr/bin/vmtoolsd TimeoutStopSec=5 PrivateTmp=yes

[Install] WantedBy=multi-user.target

oliverkurth commented 6 years ago

Thanks @meeas for reporting. Which exact version of the open-vm-tools package from Debian do you have installed? (Find out with dpkg -l open-vm-tools). We suspect that there is a patch missing.

bzed commented 6 years ago

10.2.0 will be available in unstable in a few hours. 10.1.15 is in testing.

@meeas could you give 10.2.0 a try and if the bug is still there, please let me know. (also, reporting such bugs in the debian bugtracker would be the appropriate place!)

jrespeto commented 6 years ago

I saw this issue and tried this fix also to start vmtoolsd After=display-manager.service. Looks to have resolved my issue.

Also tested this on a fresh install of kali linux.

vmtoolsd -v
VMware Tools daemon, version 10.3.0.5330 (build-8931395)

uname -a 
Linux Kali 4.18.0-kali2-amd64 #1 SMP Debian 4.18.10-2kali1 (2018-10-09) x86_64 GNU/Linux

# cat /etc/systemd/system/multi-user.target.wants/open-vm-tools.service 
[Unit]
Description=Service for virtual machines hosted on VMware
Documentation=http://open-vm-tools.sourceforge.net/about.php
ConditionVirtualization=vmware
DefaultDependencies=no
Before=cloud-init-local.service
RequiresMountsFor=/tmp
After=display-manager.service

[Service]
ExecStart=/usr/bin/vmtoolsd
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

rhertzog commented 6 years ago

This issue seems to still exist in version 10.3.5-1 that is in Debian Testing and Kali Linux. I'm not sure that delaying the start is the best choice but there must definitely be something to reload/restart vmtoolsd once the graphical session has been started. How is it supposed to work currently ? (Last bug report on the topic in Kali https://bugs.kali.org/view.php?id=5051)

rhertzog commented 6 years ago

I can reproduce this problem but not on every boot. I think it's a race condition between the load of drm/vmwgfx kernel modules and the start of the service. When the issue is present, I can see (in the logs, configured at debug level) some lines with "Failed to open DRM render node." and "Failed to open DRM card node.". And effectively a simple restart is enough to bring back things in correct order.

My proposed fix is thus to modify the .service file by adding an ExecStartPre command loading the appropriate kernel driver:

[Service]
ExecStartPre=-/sbin/modprobe vmwgfx
ExecStart=/usr/bin/vmtoolsd
TimeoutStopSec=5

With this, I can no longer reproduce the problem and we are also not delaying it further in the systemd dependency tree. The "After: display-manager.service" directive was IMO not a good idea, it goes contrary to the logic of "Before=cloud-init-local.service" and "DefaultDependencies=no" which tries to execute this service quite early in the boot process (while display-manager is not designed to run so early).

rhertzog commented 5 years ago

@oliverkurth Can you reassess this bug in the light of the information that I provided? I don't know what patch you were referring to but the missing patch is unlikely to be the problem unless the patch is not in the last released version that I tested (10.3.5).

What do you think of my suggested fix? The only downside is possibly that you might not want to try to load that driver on a headless VM. The debian packaging even splits the graphical part (which includes the resolutionKMS plugin) in a separate package. Thus I suggested in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=915031#5 to add the "ExecStartPre" in a separate configuration file in the service "drop-in" directory (eg /lib/systemd/system/open-vm-tools.service.d/desktop.conf).

mykmelez commented 5 years ago

See https://communities.vmware.com/message/2816924 for additional reports of this issue in Ubuntu 18.10 and 16.04 LTS guests.

bzed commented 5 years ago

@oliverkurth what do you think about @rhertzog 's suggested fix?

oliverkurth commented 5 years ago

@thomashvmw can you have a look?

bzed commented 5 years ago

If the ExecStartPre=-/sbin/modprobe vmwgfx is the proper fix, then I'd suggest to add it as systemd override in the open-vm-tools-desktop package. This would ensure that it does not affact the server installation but also works for the desktop users.

thomashvmw commented 5 years ago

My preference would NOT be to force modprobe vmwgfx, since one may want to start up the system with vmwgfx disabled, and it might be pretty hard to track down what's actually loading vmwgfx. As a developer I frequently boot the system without vmwgfx to avoid bugs created during development.

We had a previous issue on this and there IIRC the conclusion was that the best would be to make sure the open-vm-tools service was launched after the service that normally loads the vmwgfx module. ("graphical?"). Let me see if I can dig up that issue.

To clarify a bit, vmwgfx does not have to be loaded, but the loaded status needs to be the same both when the system-wide vmtoolsd starts and when the user vmtoolsd starts. If it's loaded, resolutionKMS kicks in. Otherwise resolutionSet takes over.

thomashvmw commented 5 years ago

So it's actually Issue #253. The suggestion is to add After=graphical.target Under the Unit section of /etc/systemd/system/multi-user.target.wants/open-vm-tools.service

Courtesy of @pha5matis

bzed commented 5 years ago

No, that is way too late as it needs to be available for cloud-init before the network comes up.

thomashvmw commented 5 years ago

OK, so is there a way to honor the modprobe.blacklist=vmwgfx boot option with your solution or will it help to add a blacklist option to vmwgfx itself?

rhertzog commented 5 years ago

On Wed, 13 Mar 2019, thomashvmw wrote:

OK, so is there a way to honor the modprobe.blacklist=vmwgfx boot option with your solution or will it help to add a blacklist option to vmwgfx itself?

Maybe we can use "modprobe --use-blacklist vmwgfx" in the systemd unit?

It might need some testing to ensure that it honors the kernel command line as well and that it does what you want.

Cheers, -- Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/

bzed commented 5 years ago

Could somebody with a desktop in VMware please give it a try?

thomashvmw commented 5 years ago

I can confirm that "modprobe -b vmwgfx" or "modprobe --use-blacklist vmwgfx" honors the "modprobe.blacklist=vmwgfx" command line option when quickly tested on ubuntu and fedora so that should work fine.

bzed commented 5 years ago

@thomashvmw @rhertzog after reading https://www.freedesktop.org/software/systemd/man/modules-load.d.html# I think the better option would be to let systemd-modules-load.service handle this. A quick look on the systemd source gave me the impression, that this will handle the blacklist just fine.

We actually have a file there which load the fuse module, so I'll add vmwgfx.

thomashvmw commented 5 years ago

Sounds fine with me!

huornlmj commented 3 years ago

This issue is still present in Ubuntu 18.04 on VMware work station 16.1.1 build-17801498.

freemedom commented 9 months ago

It looks like the issue has been fixed, but it still appears in my Ubuntu???

And many repair commands don't work for me. I don't know much about linux. I've been working on it for six hours, and it's still unresolved...

freemedom commented 9 months ago

I spent five hours trying various things, but nothing worked. Ubuntu 18.04.5 LTS VMware® Workstation 16 Pro install in Windows 10 Home, 64-bit (Build 19042.1288) 10.0.19042 Copying and pasting between my windows and Ubuntu was fine, as was dragging of files. The only problem is the resolution, 640x480.

$ xrandr xrandr: Failed to get size of gamma for output default Screen 0: minimum 640 x 480, current 640 x 480, maximum 640 x 480 default connected primary 640x480+0+0 0mm x 0mm 640x480 73.00*

sudo ./vmware-install.pl sudo vmware-uninstall-tools.pl sudo apt-get autoremove open-vm-tools sudo apt-get install open-vm-tools open-vm-tools-desktop sudo apt install open-vm-tools-desktop -y sudo rm -rf /etc/vmware-tools sudo apt-get remove --purge open-vm-tools open-vm-tools-desktop sudo apt-get install --reinstall open-vm-tools sudo apt-get install --reinstall open-vm-tools-desktop sudo gedit open-vm-tools.service systemctl restart open-vm-tools.service sudo gedit /etc/systemd/system/multi-user.target.wants/open-vm-tools.service systemctl status open-vm-tools vmtoolsd -v systemctl daemon-reload systemctl restart open-vm-tools ps awx | grep vmtoolsd sudo apt purge open-vm-tools sudo apt update sudo apt install open-vm-tools open-vm-tools-desktop sudo gedit /lib/systemd/system/open-vm-tools.service systemctl restart vmtoolsd

search Truncated maximum resolution for register modes to VRAM size: VRAM=8388608 bytes, max wh(1672, 1254) in .log file

add svga.minVRAMSize=16777216 svga.minVRAM8MB=TRUE in .vmx file

dpkg -s open-vm-tools Package: open-vm-tools Status: install ok installed Priority: optional Section: admin Installed-Size: 2093 Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com Architecture: amd64 Version: 2:11.0.5-4ubuntu0.18.04.3 Replaces: open-vm-tools-desktop (<< 2:10.0.7-3227872-2~)

/var/log/vmware-vmsvc.log /var/log/vmware-vmsvc-root.log [2024-02-22T12:03:23.269Z] [ message] [vmsvc] Log caching is enabled with maxCacheEntries=4096. [2024-02-22T12:03:23.270Z] [ message] [vmsvc] Core dump limit set to -1 [2024-02-22T12:03:23.270Z] [ message] [vmtoolsd] Tools Version: 10.3.10.11 (build-13959562) [2024-02-22T12:03:23.605Z] [ message] [vmsvc] Cannot load message catalog for domain 'hgfsServer', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.607Z] [ message] [vmtoolsd] Plugin 'hgfsServer' initialized. [2024-02-22T12:03:23.607Z] [ message] [vix] QueryVGAuthConfig: vgauth usage is: 1 [2024-02-22T12:03:23.607Z] [ message] [vmsvc] Cannot load message catalog for domain 'vix', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.607Z] [ message] [vmtoolsd] Plugin 'vix' initialized. [2024-02-22T12:03:23.607Z] [ message] [vmsvc] Cannot load message catalog for domain 'autoUpgrade', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.607Z] [ message] [vmtoolsd] Plugin 'autoUpgrade' initialized. [2024-02-22T12:03:23.607Z] [ message] [vmsvc] Cannot load message catalog for domain 'deployPkg', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.607Z] [ message] [vmtoolsd] Plugin 'deployPkg' initialized. [2024-02-22T12:03:23.685Z] [ message] [vmsvc] Cannot load message catalog for domain 'grabbitmqProxy', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.685Z] [ message] [vmtoolsd] Plugin 'grabbitmqProxy' initialized. [2024-02-22T12:03:23.685Z] [ message] [vmsvc] Cannot load message catalog for domain 'guestInfo', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.685Z] [ message] [vmtoolsd] Plugin 'guestInfo' initialized. [2024-02-22T12:03:23.685Z] [ message] [vmsvc] Cannot load message catalog for domain 'powerops', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:23.685Z] [ message] [vmtoolsd] Plugin 'powerops' initialized. [2024-02-22T12:03:23.689Z] [ message] [resolutionCommon] resolutionCheckForKMS: dlopen succeeded. [2024-02-22T12:03:24.144Z] [ warning] [resolutionCommon] resolutionCheckForKMS: No system support for resolutionKMS. [2024-02-22T12:03:24.144Z] [ message] [vmsvc] Cannot load message catalog for domain 'timeSync', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:24.144Z] [ message] [vmtoolsd] Plugin 'timeSync' initialized. [2024-02-22T12:03:24.144Z] [ message] [vmsvc] Cannot load message catalog for domain 'vmbackup', language 'zh', catalog dir '/etc/vmware-tools'. [2024-02-22T12:03:24.144Z] [ message] [vmtoolsd] Plugin 'vmbackup' initialized. [2024-02-22T12:03:27.149Z] [ warning] [vmsvc] SimpleSock: failed to connect (1022 => 976), error 110: 连接超时 [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'vmbackup'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'timeSync'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'powerops'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'guestInfo'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'grabbitmqProxy'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'deployPkg'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'autoUpgrade'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'vix'. [2024-02-22T12:08:02.671Z] [ message] [vmtoolsd] Unloading plugin 'hgfsServer'.

https://bugs.launchpad.net/ubuntu/+source/open-vm-tools/+bug/1818473

/etc/vmware/tools.conf vmsvc.level = debug [2024-02-21T17:15:26.559Z] [ debug] [resolutionCommon] resolutionXorgDriverVersion: Scanning for VMWare Xorg drivers. [2024-02-21T17:15:26.559Z] [ debug] [resolutionCommon] resolutionXorgDriverVersion: Looking for "/usr/lib64/xorg/modules/drivers/vmware_drv.so". [2024-02-21T17:15:26.559Z] [ debug] [resolutionCommon] resolutionXorgDriverVersion: Looking for "/usr/lib/xorg/modules/drivers/vmware_drv.so". [2024-02-21T17:15:26.560Z] [ debug] [resolutionCommon] resolutionXorgDriverVersion: Driver found. Looking for version info. [2024-02-21T17:15:26.562Z] [ debug] [resolutionCommon] resolutionXorgDriverVersion: Version info found: 13.3.0 [2024-02-21T17:15:26.562Z] [ debug] [resolutionCommon] resolutionCheckForKMS: ResolutionKMS enabled based on Xorg driver version. [2024-02-21T17:15:26.563Z] [ message] [resolutionCommon] resolutionCheckForKMS: dlopen succeeded. [2024-02-21T17:15:26.767Z] [ debug] [resolutionCommon] resolutionDRMRPrimaryCheckOpen: Failed to open DRM render node. [2024-02-21T17:15:26.884Z] [ debug] [resolutionCommon] resolutionDRMRPrimaryCheckOpen: Failed to open DRM card node. [2024-02-21T17:15:26.885Z] [ warning] [resolutionCommon] resolutionCheckForKMS: No system support for resolutionKMS.

This problem is caused by my own modification of the files in VMware Workstation. (￣▽￣) I forgot that I ran a program a year ago to modify some files related to preventing virtualization detection in C:\Program Files (x86)\VMware\VMware Workstation\x64 directory. I have just reinstalled/repaired VMware Workstation 16 and the resolution Autofit Guest is back to normal.

vmware / open-vm-tools

Service starts too early to find resolutionKMS in Debian Testing #214