search

microsoft / WSL

Issues found on WSL
https://docs.microsoft.com/windows/wsl
MIT License
17.49k stars 822 forks source link

Vmmem high CPU usage #6982

Open manuelserradev opened 3 years ago

manuelserradev commented 3 years ago

Windows Build Number

Microsoft Windows [Version 10.0.21387.1]

WSL Version

Kernel Version

5.10.16

Distro Version

Ubuntu 20.04

Other Software

No response

Repro Steps

I'm struggling to actually repro this issue deterministically. But looks like more prominent when waking up from sleep. Whit this issue I'd like to gather feedback on how to collect the most useful information to debug this behavior.

I tried to isolate the failing component (ex. not running Docker Desktop and WSL2 "only"), but I failed miserably.

Usually my workflow:

In the past I tried to alleviate the issue with a .wslconfig but didn't worked.

My machine (Lenovo X1 Carbon) has 16GB of RAM and usually Vmmem RAM consumption float around ~4GB.

Excluded those hiccups WSL2 experience is pretty neat and flowlessy.

Expected Behavior

WSL2 stay idle when no user workload are launched.

Actual Behavior

Vmmem "randomly" uses high cpu/power amount (60%-70%) for couple of minutes (2 to 5 min) before settling down. This also happen when on battery without doing anything WSL2 releated (but with Docker Desktop running), killing autonomy.

Diagnostic Logs

I'll provide an .etl dump as the next hiccup occurs.

manuelserradev commented 3 years ago

Today, after wake up from sleep I managed to repro.

wsl.zip

Matsemann commented 3 years ago

I also see high vmmem cpu usage from time to time (stuck at ~25% CPU). When checking htop on my only running image, I see that one core is being used 100%, but none of the processes are using any CPU. So not sure what's happening / what's the culprit. Docker images not running at the moment.

C:\Users\mats>wsl -l -v
  NAME                   STATE           VERSION
* Ubuntu-20.04           Running         2
  docker-desktop-data    Stopped         2
  docker-desktop         Stopped         2

Have the opposite problem with docker, though. Suddenly it will use all available ram even when not running any containers (but the wsl distributions are running)

Edit: A pic of htop when this occurs. Nothing visible using CPU, but it's using much CPU on multiple cores anyway. image

manuelserradev commented 3 years ago

The day after, also waking up from night sleep.

Like @Matsemann htop do not show anything relevant.

wsl2.zip

ll931217 commented 3 years ago

The day after, also waking up from night sleep.

Like @Matsemann htop do not show anything relevant.

wsl2.zip

The high CPU usage can be seen from Task Manager

Matsemann commented 3 years ago

Yes, and from the htop screenshot. But you cannot see what is using the CPU (other than the generic vmmem but that can be anything). So there is something spinning somewhere draining resources for no good reason.

ChrisJefferson commented 3 years ago

I'm hitting this too, so if anyone has any suggestions on how to capture what is going on, I am happy to record it.

ChrisJefferson commented 3 years ago

Update, I can reproduce this 100% of the time by turning off my 2nd monitor (one core in Linux gets pegged to 100%, but htop doesn't show any processes using CPU power).

If I run 'xeyes', as soon as xeyes open the mystery CPU usage disappears, so I assume it's something to do with the new graphics integration, and actually running a program gives it a kick?

Matsemann commented 3 years ago

Great find! I've always experienced it after computer has been sleeping (like when coming back after lunch), but I also can reproduce it by reconnecting a monitor. Which I guess is what happens when the computer starts after sleep.

Running xeyes or any wslg app doesn't seem to stop it here, though.

Matsemann commented 3 years ago

I'm hitting this too, so if anyone has any suggestions on how to capture what is going on, I am happy to record it.

Yes, if someone from wsl team could tell us how to give them better diagnostics that would be nice. As we've said, nothing shows up as using resources in the wsl images themselves. But we can see a high cpu core usage in htop, or vmmem cpu usage in task manager. But don't know how to see what is causing it.

fmiqbal commented 3 years ago

Update, I can reproduce this 100% of the time by turning off my 2nd monitor (one core in Linux gets pegged to 100%, but htop doesn't show any processes using CPU power).

Can confirm,

  1. wsl --shutdown, set monitor projection to single screen
  2. Startup wsl, normal htop
  3. Make monitor projection to extend
  4. Suddenly 1 core spin up in htop

I tried messing with refresh rate (my first monitor has 144, my second 60), but no luck. In the process, I found that when I tried to change the refresh rate, the screen "refresh", then the 100% core in htop gone. So I tried to just reset the graphic driver (Win+Ctrl+Shift+B), and the 100% core is also gone.

Although, currently its fresh restart and reproduced reliably, but only 1 core spinning up. In the past I had like 3 core spinning up out of nowhere. I also use Docker for Windows, which for this purpose I turn off first, because I also had bad experience on that docker hogging CPU while no container running

Update Using docker for windows, I get 3 core spinning up when projecting to second screen, and Win+Ctrl+Shift+B still works like a charm

Windows 10, build 21390.2025 WSL2, Arch Linux, 2 monitor setup,

onomatopellan commented 3 years ago

I think this is indeed related to WSLg, which creates a hidden "system distro" instance for each running user distro. That's why you can't see anything in htop. When the 100% core usage happens you can run this from the terminal: wsl.exe --system -d $WSL_DISTRO_NAME top

I bet the culprit is the weston binary inside the system distro. Recently there was a similar problem that has been fixed in the weston repository.

joewood commented 3 years ago

As a possible confirmation of this, I had this issue continuously (with vmmem using 50% of CPU without anything running on WSL2). I disabled WSLg in wslconfig and restarted the distributions. The issue (so far) seems to be fixed.

onomatopellan commented 3 years ago

With wsl.exe --update you will get WSLg 1.0.24 which contains some fixes about high cpu usage.

joewood commented 3 years ago

If it helps, I was already running 1.0.24 (updated yesterday) and still seeing the issue. Disabling WSLg seems to have fixed it. I'll keep testing.

lishan89uc commented 3 years ago

I too can reproduce this 100% by turn on/off my second monitor. Is this issue user "workaround-able" or do we have to wait for Microsoft. If so is there a way to expedite such a serious issue?

Side note: opening xeyes only drop the CPU usage from 25% to 17%. and after I closed xeyes the cpu went back up to 25%. So far I have not found a reliable way to make it run normally again other than to disable wslg with guiApplications=false added to .wslconfig inside the personal folder, do a complete wsl shutdown and restart, or wait it out. If anyone find a better way to keep the functionality (preferably automatable) as a workaround that be great.

fmiqbal commented 3 years ago

@lishan89uc have you tried Win+Ctrl+Shift+B

GoaLitiuM commented 3 years ago

I occasionally get this issue as well and it usually gets fixed when I shut down the WSL completely. This time I tried pressing the Win+Ctrl+Shift+B combo and that seems to have fixed the CPU usage in this instance.

manuelserradev commented 3 years ago

I also experienced this today and the Win + Ctrl + Shift + B trick solved the issue.

Worth mentioning that this is now happening on Windows 11:

Edition Windows 11 Pro
Update  21H2
Installed on    ‎01-‎Jul-‎21
OS build    22000.51
Experience  Windows Feature Experience Pack 421.16300.0.3
jamestut commented 3 years ago

Also happened on mine. Consistently happens after locking the screen. My system is a laptop with internal 4K display (switched off) and external 5K display via Thunderbolt (set to second screen only). Build 22000.71, WSL2 kernel version 5.10.43, WSL2 last updated 18 July 2021.

letrhee commented 3 years ago

When vmmem's CPU usage is very high, in my case, stopping Docker desktop solved the problem.

addshore commented 3 years ago

I occasionally get this issue as well and it usually gets fixed when I shut down the WSL completely. This time I tried pressing the Win+Ctrl+Shift+B combo and that seems to have fixed the CPU usage in this instance.

This seems to work for me as a temporary fix.

I'v see this in a few different states, either 1 CPU core or 3 CPU cores all at 100% and nothing showing up in top etc as actually using CPU.

Edition Windows 11 Pro
Version 21H2
Installed on    ‎29/‎07/‎2021
OS build    22000.100
Experience  Windows Feature Experience Pack 421.18901.0.3
jibbers42 commented 3 years ago

If you don't need gui apps then you can disable wslg while waiting for a fix. I did it a couple days ago and it seems to have stopped the high-cpu-after-sleep issue.

Add the following to <USERPROFILE>\.wslconfig and restart the machine (or maybe wsl --shutdown is good enough).

[wsl2]
guiApplications=false

See https://github.com/microsoft/wslg#wslg-system-distro

Mystic8b commented 3 years ago

I do not use sleep mode on my office PC, it is always on. And I have the same problem. Perhaps it appears after connecting via RDP, but I cannot confirm this yet. @jibbers42 tx man, i'll try

benhillis commented 3 years ago

Could somebody who is hitting this provide the output of dmesg? I suspect this is WSLg-related.

ctataryn commented 3 years ago

Hi folks, I was having issues with vmmem causing my laptop fans to kick in after I had installed Microsoft PowerToys when I needed to do some batch image resizing. If I quit "Microsoft PowerToys v0.43.0" from the task tray then vmmem went back to normal.

I hope this is relevant and helps someone else in a similar predicament.

Mystic8b commented 3 years ago

I also use power toys, but at the moment the cpu load is normal and cannot confirm. Interesting)

ctataryn commented 3 years ago

I also use power toys, but at the moment the cpu load is normal and cannot confirm. Interesting)

Another thing I notice is that vmmem won't actually go away until I issue another wsl --shutdown after quitting PowerToys.

Steps:

1) Quit Docker Desktop from task tray 2) from cmd.exe: wsl --shutdown 3) Observe vmmem is still running as a process and taking about 5% CPU 4) Quit PowerToys from task tray 5) Observe vmmem is still running, however CPU usage is 0% 6) from cmd.exe, again issue: wsl --shutdown 7) Observe that vmmem is no longer present in the Task Manager

fmiqbal commented 3 years ago

@ctataryn sorry, can't confirm. As soon as I shutdown the wsl, vmmem is gone image

ctataryn commented 3 years ago

@fmiqbal what version of PowerToys? I should add two things:

1) The name of PowerToys that I have in my Start Menu is "PowerToys (preview)" 2) There is a secondary task tray icon that I don't see listed on your machine that I have called "PowerToys Awake" (blue tea cup) image

fmiqbal commented 3 years ago

@ctataryn you can see on my last screenshot its v0.43.0, my PowerToys Awake is disabled (I tried with it enabled, same). I think that might be different issue? so far this issue only mention multi-monitor/projection and wslg

nbdd0121 commented 3 years ago

I don't think PowerToys is relevant. I've hit this issue before but I don't have PowerToys installed. The weston theory is correct IMO.

elgerm commented 3 years ago

So, I think the status needs repro can be changed right? After sleep vmmem starts eating the cpu, then resetting the graphic driver (win + ctrl + shift + b) stops it.

Oyami-Srk commented 3 years ago

wsl --system -d Ubuntu top shows the high cpu usage is caused by weston. UV(B_U}LNE7`KEPA)VQXO0K

and it not even just high cpu usage on my machine. The worst part of it is taking my input focus away to mstsc.exe. I installed PowerToy but has already killed it. I will try to install the latest commit of microsoft/weston-mirror otherwise I have to disable WSLg for the normally using of my computer.

billbarni commented 3 years ago

Even after stopping Docker, all running images, Vmmem still eats 4GB+ RAM in my machine. If I restart the service it goes back to "normal" ~1GB+ RAM.

This problem is going for AGES without a fix. Hyper-V was bad... and WSL2 is...? Bad.

(I changed back Docket to Hyper-V, uninstalled everything WSL2 related and most of my issues just vanished)

vattay commented 3 years ago

I work with WSL2 every day and love it, but my hands are getting tired from using (win + ctrl + shift + b) dozens of times per day.

elgerm commented 3 years ago

My guess is the guys making wsl2 aren't actually using it

Mystic8b commented 3 years ago

My guess is the guys making wsl2 aren't actually using it

I have these thoughts from almost all products, not just MS. Otherwise I cannot explain it, really.

GoaLitiuM commented 3 years ago

I have not hit this issue anymore after disabling WSLg following these steps;

If you don't need gui apps then you can disable wslg while waiting for a fix. I did it a couple days ago and it seems to have stopped the high-cpu-after-sleep issue.

Add the following to <USERPROFILE>\.wslconfig and restart the machine (or maybe wsl --shutdown is good enough).

[wsl2]
guiApplications=false

See https://github.com/microsoft/wslg#wslg-system-distro

Christoph-AK commented 3 years ago

I have this issue with Docker Desktop. I am using a mobile Lenovo Ideapad with Intel 1165G7 with Iris Xe and NVidia Mx450 GPUs. I suspect this might be a driver issue with multiple GPUs. Installing the latest Intel and Nvidia drivers and reinstalling WSL-Ubuntu seems to have fixed the vmmem-cpu usage problem for me for the time being (allthough hardware accelaration in some microsoft apps like excel and teams broke - welcome to 2021...)

lukasamd commented 3 years ago

I had this issue with Docker Desktop + phpStorm IDE and project inside WSL (Ubuntu 20.04 from store). It happened sometimes once a day, sometimes more often. It wasn't big deal, because it's easy to kill such process. The bigger issue was missing git connection - phpStorms sometimes "lost" information about modified files and I had to close and reopen IDE. I think it may be related, because that 100% CPU usage sometimes occurred after some checkout/pull/push/other git operation and remained until I kill process.

Resolved that by running phpStorm from WSL - tried W11 and WSLg, but it has some issues with windows focus and also overall I was not happy with W11. Right now use W10 and phpStorm inside WSL + X410 for X Server and after almost two weeks:

I think it's something related to communication between host and WSL filesystem / processes

galiovsky commented 3 years ago

@benhillis Unfortunately, I couldn't find anything in dmesg output. In weston.log, just a couple of ordinary messages caused by DisplayLayoutChange (the issue starts at 08:20:43.834).

I've been hitting this issue for many months and can reproduce it all the time on my laptop. Nothing has to be running in the WSL2 distro, it can be freshly booted. Doesn't matter whether it's Fedora or Ubuntu. The moment I attach or detach an external display while the distro is running, Vmmem starts to load one CPU core for 100 %, although no load is visible "inside" the distro itself. Please let me know if I can be of any assistance in troubleshooting this. Thank you!

Windows 11 Pro 10.0.22000.282, Ryzen 4650U integrated graphics, both screens 1920x1080, one scaled at 125 %, the other 100 % wsl.etl.zip

galiovsky commented 3 years ago

Hey everyone, try out the WSL Preview version (available from Microsoft Store or from this repository), it seems to have fixed the issue for me!

ziruizhuang commented 3 years ago

wsl --system -d Ubuntu top shows the high cpu usage is caused by weston. and it not even just high cpu usage on my machine. The worst part of it is taking my input focus away to mstsc.exe. I installed PowerToy but has already killed it. I will try to install the latest commit of microsoft/weston-mirror otherwise I have to disable WSLg for the normally using of my computer.

Similar here with WSL Kernel version: 5.10.60.1, showing weston consuming CPU clocks. Windows 11 22000.282 with Intel 9700K processor / two monitors connected to NVIDIA 2080 Super graphic card.

The CPU consumption by Vmmem seems to be random and I haven't find a way to reproduce.

2021-10-26 (1)

galiovsky commented 3 years ago

@ziruizhuang Disable WSLg if you don't need it. If you do need it, check /mnt/wslg/versions.txt. If the versions are lower than https://github.com/microsoft/wslg/issues/528#issuecomment-948920724, try updating to https://github.com/microsoft/WSL/releases/download/0.48.2/Microsoft.WSL_0.48.2.0_x64_ARM64.msixbundle.

JoschD commented 3 years ago

Hey everyone, try out the WSL Preview version (available from Microsoft Store or from this repository), it seems to have fixed the issue for me!

As soon as I installed this, WSLg became unusable for me. I am running pycharm from Ubuntu and while I was doing nothing the CPU from VmmemWSL was around 0% but shot up to 50-80% as soon as I started typing anything and everything lagged. Keystrokes took many milliseconds to seconds to be performed. Not sure where this problem comes from. It doesn't seem to subside, even after uninstalling the WSL2 from the store again.

ayon06 commented 3 years ago

Surface Pro X2 running Ubuntu

I am also seeing this issue after trying to re-access Ubuntu file system, after sleeping and waking up my cpu, via VS Code. VS Code will try for 3 minutes to connect to Ubuntu, then provide a "cannot connect" error. vmmem will continue to take up 25%+ CPU until I manually shut down WSL

Edit: It happens on successful reconnections as well. vmmmem jumps up as much as 40%+. Jumps back down a few minutes later. This data is with @jibbers42 suggested fix

greg-is-kub commented 3 years ago

build : asus pro 15 oled windows insider dev channel Version 10.0.22000 Build 22000 running wsl 2 on Ubuntu 18.04 also having this issue I am not able to put my computer to sleep or he will choke on his fans and overheat to death

Doesn't matter if I use wsl or not, it always eats 2 Go of RAM minimum and up to 4/5Go At first I thought I had a pirat virtual machine mining crypto on my computer

the win+ctrl+shift+B trick doesn't change much for me

galiovsky commented 3 years ago

As soon as I installed this, WSLg became unusable for me. I am running pycharm from Ubuntu and while I was doing nothing the CPU from VmmemWSL was around 0% but shot up to 50-80% as soon as I started typing anything and everything lagged. Keystrokes took many milliseconds to seconds to be performed. Not sure where this problem comes from. It doesn't seem to subside, even after uninstalling the WSL2 from the store again.

@JoschD what's the content of your current /mnt/wslg/versions.txt?

BTW, the WSL support in recent PyCharm Pro versions is pretty decent (save for the slow environment indexing but they know about this) so there's not much reason to run it via WSLg, though WSL support is not included in the PyCharm Community edition IIRC 😥

galiovsky commented 3 years ago

Surface Pro X2 running Ubuntu

I am also seeing this issue after trying to re-access Ubuntu file system, after sleeping and waking up my cpu, via VS Code. VS Code will try for 3 minutes to connect to Ubuntu, then provide a "cannot connect" error. vmmem will continue to take up 25%+ CPU until I manually shut down WSL

Edit: It happens on successful reconnections as well. vmmmem jumps up as much as 40%+. Jumps back down a few minutes later. This data is with @jibbers42 suggested fix

@ayon06 Then it's likely a different issue than I had. Re-reading the original report carefully now, the load in my case never settled down, but the container was behaving more or less normally (no timeouts, etc).

JoschD commented 3 years ago

@galiovsky I did a full WSLg uninstall and reinstalled via terminal (not via marketplace, so the process is Vmmem not VmmemWSL) and it seems to work fine now. What's new now is, that I am on the 5.10.60.1-microsoft-standard-WSL2 kernel. Lately I don't even have problems moving windows between monitors (which was a problem in the past, I think because of different scaling).

I feel also the super-high CPU usage after sleep has gotten better, I think. Might be placebo, but I think whatever the dev's are doing, they are going in the right direction :)

My /mnt/wslg/versions.txt now reads

WSLg ( x86_64 ): 1.0.26+Branch.main.Sha.26ce2c09b86442f3c7f4f6462f770ed2afa76a25
Mariner: VERSION="1.0.20210224"
FreeRDP: b05321cd4e6a862aef76163a69db4e1910245736
weston: 46756d0e77e5c01b5995fbbee6f3ab0db9b30612
pulseaudio: 2f0f0b8c3872780f15e275fc12899f4564f01bd5
mesa:

So I am on the version, that gave you problems. I will keep an eye out and try to maybe upgrade to 1.0.29 when I have time to handle the potential fallout.

Also, thank you very much for the info about the pycharm WSL support! I will have a look. If it works similarly well as VSCode, that would be indeed a great solution (the indexing in pycharm is slow af in any case ;) ).