Closed nils-acumen closed 3 months ago
nils-acumen
commented
2 years ago Additional information: I had docker idling around for like half a day only using my other container (a sql dev version) to do some other work. It just started successfully without restarting pc or docker.
Event logs showing 2 more errors:
and
Which is interesting since I never had the whole username/ password issue. Maybe its related to this issue: https://github.com/microsoft/navcontainerhelper/issues/1321
freddydk
commented
2 years ago 8G is the default memory limit. How much memory do you have on the host computer?
freddydk
commented
2 years ago Did a quick test on my machine (which has 128G)
I created a hyperv container with 8G and had that running for a bit
Then I created a container with 32G (same name, which causes containerhelper to remove the old and create a new container)
That results in the following resource monitor output:
the container removal and creation happens in the 3rd quadrant.
As you can see - the physical memory used on the host doesn't change whether you use 8G or 32G
but the commit charge does.
Docker/ContainerD will allocate the 32G in the paging file and reserve that for the container - but you cannot just set the paging file to 10000G with a 16G machine. I am not sure about the details of this - but my understanding is that when containerD gives the resource in use error - it is because it is unable to allocate sufficient space in the paging file for the container.
Needless to say, I have never seen the resource in use with my 64Gb or my new 128Gb machine.
freddydk
commented
2 years ago If your computer is low on memory, process isolation might be better (but sometimes that has other problems on desktop OS's)
nils-acumen
commented
2 years ago Ok that makes sense. I've got only 16GB memory on my machine so maybe it will go away when I upgrade to more?
I've tried to just recreate the bc container with a higher memory limit: Script
$containerName = 'bcserver'
$password = 'P@ssw0rd'
$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$credential = New-Object pscredential 'admin', $securePassword
$auth = 'UserPassword'
$artifactUrl = Get-BcArtifactUrl -type 'Sandbox' -country 'nz' -select 'Latest'
New-BcContainer `
-accept_eula `
-containerName $containerName `
-credential $credential `
-auth $auth `
-artifactUrl $artifactUrl `
-isolation hyperv `
-updateHosts `
-memoryLimit 12G `
-alwayspullSomehow that ended in the same error as just starting the image was: Error Details
DockerDo : docker: Error response from daemon: hcsshim::CreateComputeSystem aa6c578e851e887e3b957bf68b3a1e3c161135456e248f95e44118af09e532dd: The requested resource is in use. ExitCode: 125
At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\3.0.11\ContainerHandling\New-NavContainer.ps1:1852 char:15 + ... if (!(DockerDo -accept_eula -accept_outdated:$accept_outdated - ... +
~~~~~~~~~~~~~
- CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
- FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerDo
Full log:
PS D:\bc-development> .\new-container-wiz.ps1
BcContainerHelper version 3.0.11
BcContainerHelper emits usage statistics telemetry to Microsoft
PS D:\bc-development> .\new-container-wiz.ps1
BcContainerHelper is version 3.0.11
BcContainerHelper is running as administrator
Hyper-V is Enabled
UsePsSession is True
Host is Microsoft Windows 10 Enterprise - 21H2
Docker Client Version is 20.10.17
Docker Server Version is 20.10.17
Removing entries from hosts
Removing bcserver from container hosts file
Removing bcserver-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\bcserver
Downloading application artifact /sandbox/20.3.42673.44004/nz
Downloading C:\Users\Nils\AppData\Local\Temp\8338f773-836d-499a-935c-0d44389cb24d.zip
Unpacking application artifact to tmp folder using Expand-Archive
Downloading platform artifact /sandbox/20.3.42673.44004/platform
Downloading C:\Users\Nils\AppData\Local\Temp\c86f5621-ab29-4cab-aed4-03bdabbfd72c.zip
Unpacking platform artifact to tmp folder using Expand-Archive
Downloading Prerequisite Components
Downloading c:\bcartifacts.cache\sandbox\20.3.42673.44004\platform\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
Downloading c:\bcartifacts.cache\sandbox\20.3.42673.44004\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Downloading c:\bcartifacts.cache\sandbox\20.3.42673.44004\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Fetching all docker images
Fetching all docker volumes
INFO: Windows 10 21H1/21H2 images are not yet available, using 2004 as these are found to work better than 20H2 on 21H1/21H2
Pulling image mcr.microsoft.com/businesscentral:10.0.19041.1415
10.0.19041.1415: Pulling from businesscentral
Digest: sha256:aa9f9d0779654ab9dba736a43141591a70fa419c4a9a57fd8a517086841280ff
Status: Image is up to date for mcr.microsoft.com/businesscentral:10.0.19041.1415
mcr.microsoft.com/businesscentral:10.0.19041.1415
Using image mcr.microsoft.com/businesscentral:10.0.19041.1415
Creating Container bcserver
Style: sandbox
Multitenant: Yes
Version: 20.3.42673.44004
Platform: 20.0.42653.43979
Generic Tag: 1.0.2.10
Container OS Version: 10.0.19041.1415 (2004)
Host OS Version: 10.0.19044.1826 (21H2)
Using hyperv isolation
Using locale en-NZ
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Files in C:\ProgramData\BcContainerHelper\Extensions\bcserver\my:
- AdditionalOutput.ps1
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container bcserver from image mcr.microsoft.com/businesscentral:10.0.19041.1415
aa6c578e851e887e3b957bf68b3a1e3c161135456e248f95e44118af09e532dd
New-BcContainer Telemetry Correlation Id: 9ff685cb-ec15-4be8-96d2-6ddf5cf621c1
DockerDo : docker: Error response from daemon: hcsshim::CreateComputeSystem
aa6c578e851e887e3b957bf68b3a1e3c161135456e248f95e44118af09e532dd: The requested resource is in use.
ExitCode: 125
Commandline: docker run --volume "c:\bcartifacts.cache:c:\dl" --label nav= --env isBcSandbox=Y --label
version=20.3.42673.44004 --label platform=20.0.42653.43979 --label country=NZ --env
artifactUrl=https://bcartifacts.azureedge.net/sandbox/20.3.42673.44004/nz --env multitenant=Y --env licenseFile=""
--name bcserver --hostname bcserver --env auth=NavUserPassword --env username="admin" --env ExitOnError=N --env
locale=en-NZ --env databaseServer="" --env databaseInstance="" --volume
"C:\ProgramData\BcContainerHelper:C:\ProgramData\BcContainerHelper" --volume
"C:\ProgramData\BcContainerHelper\Extensions\bcserver\my:C:\Run\my" --isolation hyperv --restart unless-stopped --env
filesOnly=False --memory 12G --env enableApiServices=Y --env useSSL=N --volume
"c:\windows\system32\drivers\etc:C:\driversetc" --env securePassword=76492d1116743f0423413b16050a5345MgB8AEUAVQAzADcAdQ
AzADcAQQBCAHgAOQBLAEIAaQBlAE0AdgA5ADQATAA4AGcAPQA9AHwAZABlADEANwBjAGYANwA0ADQANwA5AGIAYwBiAGYAMQA0ADIAMAA5ADgAYgA4AGMAN
wA2AGEAZgA4ADMAOQA1AGIAOQBmADcAZgA5AGUAZgBjADUAMQA3ADgAZQA3ADEAMgBiADkAOAAxAGEANAAzAGUANgBlADYANQBjAGQAYQA= --env
passwordKeyFile="c:\run\my\aes.key" --env removePasswordKeyFile=Y --env accept_eula=Y --env accept_outdated=Y --detach
mcr.microsoft.com/businesscentral:10.0.19041.1415
At C:\Program Files\WindowsPowerShell\Modules\BcContainerHelper\3.0.11\ContainerHandling\New-NavContainer.ps1:1852
char:15
+ ... if (!(DockerDo -accept_eula -accept_outdated:$accept_outdated - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerDoI tried using process isolation before but that only gave me errors. I believe the windows version does not match with well with the image or something similar.
Here is the windows version:
And this is what I set my paging file to (maybe thats not correct either?):
freddydk
commented
2 years ago Yeah - I would expect that.
16Gb is not a lot when running docker. if you upgrade to 32gb, the problem will be less frequent (maybe go away) and with 64gb I am pretty sure the problem will go away.
KristofKlein
commented
1 year ago our latest dev machines have 64 Gb - and it is not gone. I do have some theory about this issue...it most likely circles around the usage of Docker Desktop and goes a little bit like this:
Windows: "oh lets start my services...oh well, Dockerd? ok, lets start" DockerD: "oh boy, I have to work again? ok, let's see: there is a BC Container that wants to run automatic? fine...lets start" BCContainer: "Woohoo! Lets start! SQL, Check, St, CHeck, IIS, CHeck ..." - BC Container is UP (Fun Fact: if you now try to connect to https://Container/BC you can actually connect to it even without Docker Desktop started... so: ) Docker Dekstop: "Muuhahaha, lets get control over this DockerD - I am the one to rule it - nobody else.... wait... it is running already! How can you!!! DockerD - your master is here and tells you: Stop everything!" DockerD: "oh crap, that bully again. I better do what he told me" BCContainer: "What? stopping? boy that will take some time, but ok...." DockerD: "oh boy that takes really long...too long, Master will be not happy about this..." Docker Desktop: "How hard can it be...start/stop/startstopping/...arrghh "Docker Desktop failed to Stop (while actually starting it)" so lets wait....wait...wait....wait (clumsy BCContainer finally managed to stop everything) ... Docker Desktop resurrects (aka. you hit restart 123 times) from the dead and tells again "DockerD - I am back! Go do your work!" DockerD: "oh boy, I have to work again .... BCContainer:" Wooohooo" DockerD and Docker Desktop are both happy they managed the job....
Well, now comes HyperV ... and makes the whole topic of the container a story of it's own as it treats it more or less like a normal VM going up and down.... if now Docker Desktop or DockerD is too hard on this "timelimits of operations" it will collide with HyperV still in charge of doing things (starting/stopping) which eventually lead to the message: resource in use...
What I want to express:
does this comment help? maybe not - but I had the need to express myself :D
freddydk
commented
1 year ago Thanks @KristofKlein - I had fun reading your "story". Personally, I use Docker Engine and I don't have this problem (but I also have 128Gb) The script I use for installing/updating docker engine is here: https://github.com/microsoft/nav-arm-templates/blob/master/InstallOrUpdateDockerEngine.ps1
You could try to see whether it indeed is a Docker Desktop problem.
mc-nv
commented
8 months ago I have a similar issue to the described. I have 4 nodes with 64 GiB RAM capacity each.
All uses same OS image 10.0.19042.
Two of them start facing issue after re-imaging one year ago. Third one just bring this problem recent also after re-imaging.
PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!
Describe the issue BC container mostly not starting up. After a couple of host reboots and docker desktop restarts it sometimes may start up the BC container. When creating a new container with the script below it always comes up after creation. Problem only starts after docker restart.
Scripts used to create container and cause the issue Got this script mostly from the wiz. I've added the memory limit. My understanding was that 4G is the default (not sure if this is accurate).
Full output of scripts
Screenshots Screensshots from the Windows Event Log
Additional context
What I've tried so far
I'm really at the end of my ideas so hoping someone here has a helping idea.
Thanks