search

vmware / open-vm-tools

Official repository of VMware open-vm-tools project
http://sourceforge.net/projects/open-vm-tools/
2.26k stars 427 forks source link

PipeWire and vmtoolsd stuck using CPU even though no sounds playing (Fedora guest Windows 10 host) #516

Closed hermidalc closed 3 years ago

hermidalc commented 3 years ago

Not sure why this periodically happens, but the pipewire process will sometimes start using a lot of CPU (even though no sounds are playing) and it takes with it the vmtoolsd daemon processes. Any idea why this is happening?

top - 13:16:40 up  2:56,  1 user,  load average: 0.65, 0.68, 0.62
Tasks: 374 total,   1 running, 373 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.4 sy,  0.0 ni, 98.2 id,  0.3 wa,  0.6 hi,  0.3 si,  0.0 st
MiB Mem :  32075.2 total,  21313.4 free,   7037.2 used,   3724.7 buff/cache
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.  24276.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                   
   1435 hermida+   9 -11  345952  14844   8372 S  49.0   0.0  32:16.73 pipewire                                                                                                                                                  
    853 root      20   0  380980   8544   7392 S  16.0   0.0   6:56.30 vmtoolsd                                                                                                                                                  
   1718 hermida+  20   0  507436  37960  31364 S  10.0   0.1   7:00.16 vmtoolsd 
   ...
lemke1458 commented 3 years ago

We discussed this internally, and all we have are some wild guesses. Since pipewire also handles video, perhaps its causing some form of notifications that the resolutionKMS plugin is responding to, assuming its even running -- we should be able to tell from the logs.

There may be something interesting in the tools logs (/var/log/vmware-vmvsc /var/log/vmware-vmusr). Increasing the debug level might also show more.

Does this last long enough to try an strace to see if that gives any hints?

If you're willing to lose functionality, you can try disabling tools plugins to narrow down which is involved. The simplest thing to do is just rename them away and restart vmtoolsd. Might not be an acceptable debugging path in a real desktop though.

hermidalc commented 3 years ago

Does this last long enough to try an strace to see if that gives any hints?

PipeWire continues using high CPU with vmtoolsd until you kill the pipewire process or reboot. I still haven’t figured out when or after what the behavior starts, because it’s not doing this after booting it happens sometime after regular use of the Fedora desktop.

I’ll have to take a closer look.

hermidalc commented 3 years ago

There may be something interesting in the tools logs (/var/log/vmware-vmvsc /var/log/vmware-vmusr). Increasing the debug level might also show more.

I don't have any /var/log/vmware-vmusr-* logs how do I enable these?

hermidalc commented 3 years ago

On a possibly related note, since Fedora 34 the VMware guest sound integration hasn't been working well and sounds don't play consistently. You sometimes get the right sound and sometimes this muffled broken sound or nothing. On Fedora 33 guest and before I never had any problem on the same hardware and Windows 10 host.

The way I always test it is by adjusting the volume on the GNOME desktop multiple times. You should consistently hear the correct sound at different volume levels each time you move the slider. On Fedora 33 guest and before it always worked perfectly. On Fedora 34 it doesn't really work well the sound doesn't always play correctly and while I'm doing this test you can see the pipewire and vmtoolsd processes using the same significant CPU as mentioned in this issue, but it's not stuck when I stop changing the volume it goes away (still don't know why eventually it will become stuck and what desktop usage triggers it)

2021-06-04

So my high level guess is that this issue has to do more with sound than video?

lemke1458 commented 3 years ago

https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/tools.conf shows the all the Tools config entries, with documentation on what they do and how and why they might be modified.

Logs should be in /var/log, unless you've redirected them or these are very old Tools. Is there anything in /etc/vmware-tools/tools.conf under a [logging] section redirecting logs to a different location?

Our guess was video because there's nothing in vmtoolsd that involves audio, but we do have code watching video in order to handle desktop resize.

hermidalc commented 3 years ago

Our guess was video because there's nothing in vmtoolsd that involves audio, but we do have code watching video in order to handle desktop resize.

Why then if you run top while adjusting the Fedora 34 GNOME desktop volume meter you see the same significant CPU usage between pipewire and vmtoolsd processes as described in the OP (though it isn't stuck doing it, the CPU usage stops when you stop adjusting the volume meter)? I'm not resizing or touching the desktop size.

lemke1458 commented 3 years ago

I just installed Fedora 34 Workstation to see if I can repro, but no luck. I used a basic install, default everything. Windows 10 20H2, Workstation 16.1.2.

Started up a terminal window, ran top(1) in that, and then started moving the volume slider from the upper right.

I do see pipewire show in top, but only a few percent of CPU. gnome-shell is usually at the top. And vmtoolsd barely registers -- It shows up with .3 % CPU, then goes away -- possibly just the guestInfo loop every 30 seconds.

I did manage to get pipewire to eat up to 70% of the CPU for a few moments (1 or 2 top cycles) when I moved the mouse out out of the VM's desktop, but then it drops back to minimal.

This was with the wayland version. I swapped to Xorg and saw the same (except Xorg handled the desktop resize better).

So its likely there's something special about your config that's involved here.

Have you checked with the pipewire community?

Also, the vmtoosld logs were where they're expected to be.

hermidalc commented 3 years ago

I'm in VM guest full screen on a 4K display, maybe that makes a difference that I see more pipewire and vmtoolsd CPU usage when adjusting the volume slider up and down. I'm using Fedora 34 default everything too (Wayland,etc)

Thank you for spending the time to look at it, I did more troubleshooting to see if it was any of my running desktop apps and discovered that my OP pipewire/vmtoolsd constant CPU usage issue seems to be triggered by Atom (v1.57). I waited until it started happening and then closed running apps one at a time and only when I closed Atom did the CPU usage go away. Atom I believe is running thru XWayland so isn't Wayland native, so something with it is causing pipewire and vmtoolsd to eventually start using significant CPU. I'll see if some future version of Atom makes it go away, thank you!

hermidalc commented 3 years ago

Sorry one more comment, I'm going to try and play around with atom options --safe-mode, --clear-window-state, --enable-gpu-rasterization, and --force-gpu-rasterization to see what can circumvent the issue, but running Atom 1.57 in default mode eventually causes the CPU usage issue.

lemke1458 commented 3 years ago

Seems weird that vmtoolsd is affected by a text editor, but good to hear you've narrowed down a culprit.

hermidalc commented 3 years ago

Actually after seeing it happen and troubleshooting a few more times I believe it's Firefox on Fedora 34 not Atom. Also I failed to mention top can be a bit deceiving because it's really not using much CPU, the pipewire percentage shown is the ~50% of 1 out of 16 logical processors so like 3% total CPU.

But it's still interesting that something in Firefox can cause pipewire to get stuck using CPU and rope in vmtoolsd procs. Once you close Firefox after a few seconds (not immediately) the pipewire and vmtoolsd procs stop using CPU.

hermidalc commented 3 years ago

Actually after seeing it happen and troubleshooting a few more times I believe it's Firefox on Fedora 34 not Atom. Also I failed to mention top can be a bit deceiving because it's really not using much CPU, the pipewire percentage shown is the ~50% of 1 out of 16 logical processors so like 3% total CPU.

But it's still interesting that something in Firefox can cause pipewire to get stuck using CPU and rope in vmtoolsd procs. Once you close Firefox after a few seconds (not immediately) the pipewire and vmtoolsd procs stop using CPU.

Yep, for others who might see same thing, it's definitely Firefox. I confirmed by booting into the desktop and not running anything and even after many hours would never see the pipewire + vmtoolsd CPU behavior. Then did the same thing booted and only opened Firefox on desktop, and low and behold eventually you see the behavior and once I closed Firefox after a few seconds it would go away. Interesting that Firefox is causing some kind of video or audio interaction with pipewire and vmtoolsd. Again for reproducibility latest Windows 10 Host, latest VMware Workstation 16, latest Fedora 34 VM with all defaults GNOME Wayland, 4k display with VM in full screen mode, GNOME desktop with Firefox open.

mypetyak commented 2 years ago

Pardon the drive-by comment, but I came across this thread while trying to (re-)find some notes on a similar baseline persistent pipewire CPU usage issue related to Firefox. From what I can tell, what you're describing might be related to: https://bugzilla.redhat.com/show_bug.cgi?id=1936901

In my case, setting media.webspeech.synth.enabled: false in Firefox 95 on Fedora 35 resolved the behavior, so it might be worth checking out for what you're seeing as well.

hermidalc commented 2 years ago

@lemke1458 hopefully this is more confirmation of what I'm seeing, that when you use Firefox I see vmtoolsd and pipewire showing CPU usage even when nothing is happening in Firefox and you are on a static page without sound or anything. I'm seeing it on Fedora 35 as well like @mypetyak

lemke1458 commented 2 years ago

@lemke1458 hopefully this is more confirmation of what I'm seeing, that when you use Firefox I see vmtoolsd and pipewire showing CPU usage even when nothing is happening in Firefox and you are on a static page without sound or anything. I'm seeing it on Fedora 35 as well like @mypetyak

Thanks. Are you trying the workaround to verify its the same issue?

hermidalc commented 2 years ago

@lemke1458 hopefully this is more confirmation of what I'm seeing, that when you use Firefox I see vmtoolsd and pipewire showing CPU usage even when nothing is happening in Firefox and you are on a static page without sound or anything. I'm seeing it on Fedora 35 as well like @mypetyak

Thanks. Are you trying the workaround to verify its the same issue?

On Fedora 35 guest with the latest updates I launched Firefox, opened a few tabs of different websites while running top (the issue doesn't pop up immediately) and then waited doing nothing on Firefox until I saw vmtoolsd and pipewire CPU load. Then set media.webspeech.synth.enabled to false and restarted Firefox with the same tabs. So far I do not see the issue come back.

One a related note the issue with system and other app sounds not working that I mentioned above in this thread was indeed caused by this issue, I just didn't realize before that I had Firefox open and the issue was happening when testing the system sounds. But VM guest sounds work properly again.

hermidalc commented 2 years ago

I opened a new related issue https://github.com/vmware/open-vm-tools/issues/600. Fedora 35 and 36 VM pipewire sounds were working fine after this fix but recently after some update sounds stopped working altogether.