Docker WSL2 Windows Containers memory issue with Oracle consuming all container virtual memory no matter what the limit is set to, eventually it switches to consume pagefile on the HDD which bombs server performance. Container usage: "docker-compose up -d" with volume mapping to NTFS (C:\My\Path\To\DB\Files) and port mapping

joeltoth commented 2 years ago

Version

Windows 10 1908 and above. Win 11 not working

WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version

latest kernel update

Distro Version

Windows 10 1903/1908, Windows 11 is not working at all

Other Software

Docker Desktop 3.6.0

Docker for Windows

Using mixed mode containers with WSL2 backend (problem never occurred on WSL1 and before windows v1903)

We are using an Oracle database inside container with a local NTFS drive mapped to a folder inside a volume in the container.

We are running python in the container.

We are then running an front end site: 1) ExtJS app in one container 2) React App in another container

Repro Steps

This issue may be related to the following issue: https://github.com/docker/for-win/issues/12401

This is occurring for us in our dev environments running docker v3.6.0 and the symptoms are similar but we are running Windows containers using the WSL2 backend (mixed mode). We also have an Oracle DB with the database files stored locally in a folder in Windows and then mapped to a volume inside the container. What I have seen by doing an exec into the container is that the virtual memory used by Oracle keeps increasing until it hits WHATEVER ram limit is set. If this is set to 8 or 12GB I can watch the task manger process climb in memory usage anytime the website makes calls to the database. There is no memory being reallocated so it just keeps consuming the pagefile until it wither stops the container or completely slows my PC down. It feels like a memory leak but I am not sure. Oracle is pull/push from the container to the local files on the windows server but how could this be the cause?

This used to work in earlier docker versions before upgrading to using WSL2. I never thought this was an issue when we first upgraded so that doesn't seem like it would be the problem.

NOTE: I have seen here that Ben Dodds has had a similar issue (may not be related) recently and has reported it against docker-compose as an issue with the presence of custom port mappings: https://github.com/docker/compose/issues/8978 . No response has been given there so that was not helpful to find any sort of solution here.

Can anyone comment as this looks like we have multiple issues all pointing at poor performance in docker 3.6.0 which is not yet resolved in the latest version which also follows a new pricing model. We have tried on Windows 10 versions 1903/1908 and above. I am only ever able to up 2 containers as when I try to up the 3rd container if the memory limit on my PC is hit performance is slowed and the third container never actually comes up. In task manager each container vmemm process grows up to the memory limit of the container and sometimes will even cause the container to stop working properly.

Can anyone provide any context and direction here? Where should I report this issue if this is not the right place?

Expected Behavior

I would expect that container memory does not keep growing to WHATEVER limit I set it to.
I would expect it to do better management of memory so that it doesnt start consuming HDD space as the page file keeps growing and growing.
I would expect that I could up more than 2 containers as I was able to in prior versions of docker & windows using Hyper-V & WSL
I would expect that storing files outside the container (for client production backups) should not have a performance impact when mapped in a volume through docker-compose to a folder inside the container in linux (mixed mode)
I expect things to work as they did before with similar or better performance than when using WSL 1

Actual Behavior

All of my memory is consumed and I cannot up more than 2 containers with oracle DBs inside them mapped to volumes stored on the local server HDD.
This also happens to every developer in their local workstations so our development time has all of a sudden drastically increased.
Performance has decreased since WSL 1 and I can only use 2 containers while I used to be able to have 4+ no problem. I have 16GB RAM locally with an i7 processor and my PC has not changed on docker and windows updates have occurred.

NOTE: I am not sure if these issues are related or just red herrings: https://github.com/docker/for-win/issues/12401 https://github.com/docker/compose/issues/8978

Diagnostic Logs

Can provide whatever is needed to debug this, I am willing to work with anyone to try and get to the bottom of this as its affecting many people across the board in different situations with different database types. Docker, WSL2, Windows Containers (mixed mode) Performance issues.

What is the actual source of the issue? Does anyone really know?

romanpastu commented 2 years ago

Same issue has happened to me, I've tried all kind of configs, different distros, etc.... But it always takes massive amount of ram (no matter how much I assign), and then starts swapping which crashes my whole computer (setting swap=0 , doesn't work).

The only solution that worked for me is switching to hyper-v in docker desktop. It still consumes a massive amount of ram, but at least it doesn't start swapping and crashes.

asiridiaz commented 10 months ago

The same issue happens to me but for elasticsearch When I have a fresh start (PC Restart) everything works fine but when I Do a docker-compose down and then docker-compose up. the whole cluster is staggering and giving many errors on Timeouts and GC overheads. Also Once I restart the PC and run it works like a charm.

microsoft / WSL