Container creation gets stuck until "hosts" file is opened

[ninten]

We use an agent to create a container, install some apps, execute tests and remove the container. After some runs, the agent creation gets stuck (in a random step), and we found out that opening the file "C:\Windows\System32\drivers\etc\hosts" unblocks the process and it finishes successfully. If we don't unblock it on time, it fails and all subsequent builds will fail too until the machine is restarted and the stuck container removed manually.

This is how we create the container:

$Credential = [PSCredential]::new('username',(ConvertTo-SecureString 'password' -AsPlainText -Force));
New-BCContainer -accept_eula -assignPremiumPlan -accept_outdated -memoryLimit 12G -doNotCheckHealth -containerName 'build' -credential $Credential -image mcr.microsoft.com/businesscentral/sandbox:16.2.13509.13794-w1-ltsc2019 -restart no -shortcuts None -licenseFile 'C:\xyz\xyzLicense.flf' -includeTestToolkit -includeTestLibrariesOnly -updateHosts  -isolation hyperv  -bakFile 'C:\ProgramData\BCContainerHelper\Extensions\Backup\database.bak' 

And this is how we remove it:

Stop-NavContainer -containerName 'build'
Remove-BCContainer -containerName 'build'; Flush-ContainerHelperCache

The stack:

BcContainerHelper is version 1.0.7
BcContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Datacenter - ltsc2019
Docker Client Version is 19.03.12
Docker Server Version is 19.03.12
Fetching all docker images
WARNING: You are running specific Docker images from mcr.microsoft.com. These images will no longer be updated, you should switch to user Docker artifacts. See https://freddysblog.com/2020/07/05/july-updates-are-out-they-are-the-last-on-premises-docker-images/
Using image mcr.microsoft.com/businesscentral/sandbox:16.2.13509.13794-w1-ltsc2019
Disabling Health Check (always report healthy)
Creating Container build
Version: 16.2.13509.13794-w1
Style: sandbox
Multitenant: No
Platform: 16.0.13440.13772
Generic Tag: 0.0.9.99
Container OS Version: 10.0.17763.973 (ltsc2019)
Host OS Version: 10.0.17763.1490 (ltsc2019)
Using hyperv isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\agent\_work\_temp\xyzLicense.flf
Files in C:\ProgramData\BcContainerHelper\Extensions\build\my:
- AdditionalOutput.ps1
- license.flf
- MainLoop.ps1
- SetupNavUsers.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container build from image mcr.microsoft.com/businesscentral/sandbox:16.2.13509.13794-w1-ltsc2019

• This problem will not happen after a fresh restart.
• We use the same user to execute the agent and to manually open the hosts file.
• We managed to reproduce the problem executing the creation-stop-remove commands manually several times, but it does not happen as often as when using the agents.

[freddydk]

I am pretty sure this is caused by your antivirus software. but - if this is only for that process, you could just remove -updatehosts and avoid the hosts file update mechanism - which is mostly for when you need to access the container from a browser or vs code.

[dNsl9r]

Strange thing - get the same error here on my notebook.

Adding the -isolation hyperv parameter solves the issue for me

[freddydk]

What anti-virus software are you running - and what happens if you exclude c:\windows\system32\drivers\etc from antivirus?

[dNsl9r]

What anti-virus software are you running - and what happens if you exclude c:\windows\system32\drivers\etc from antivirus?

Kaspersky, managed by our security department. Can't exclude any files or disable the anti virus 😞

[freddydk]

Ok, pretty sure that kaspersky is the issue. If you don't use -updateHosts it probably also works - not sure if you can access the container then though. BTW - my windows 10 2020H2 actually works without updatehosts.

[dNsl9r]

Unfortunately the container is not reachable without the updateHosts - and you are miles away from the Win Version available for us 😅 Pretty sure, too that Kaspersky is the issue - didn't had the issue a week ago..

[ninten]

I am pretty sure this is caused by your antivirus software. but - if this is only for that process, you could just remove -updatehosts and avoid the hosts file update mechanism - which is mostly for when you need to access the container from a browser or vs code.

Unfortunately we tried removing our antivirus software (Windows Defender) and -updateHosts and we still get the same issue. Our only workaround has been switching to an azure Windows 10 machine with Docker Desktop and it seems to be working fine, but we would like to make it run in our local servers 😥.

[freddydk]

I don't know what local policies you have on your local servers, but if something works on a freshly installed Azure VM and not on your local server, then it is likely some local policy or software blocking things. These things can take ages to troubleshoot. I would recommend that you shift to artifacts and ensure that you are running compatible container and host OS' - that should remove most of the incompatibility issue we have seen and then you would have to disable local policies and other things until it works. Fighting these things very often really doesn't pay off - I was helping another partner, where their pipelines where running fine and then suddenly one day everything stopped. Everybody looked at windows updates or docker, but it ended up being the IT department who had enabled some security policy, which caused these things. IMO - agents should be something you never access with RDP and the machine should not be under normal desktop security settings. My agents are created using http://aka.ms/getbuildagent - and if they fail - I delete them and create a new (btw - the partner from before ended up adopting that) Another possible way is to use Azure Hosted agents.

[freddydk]

Closing this one as there is nothing really I can do to solve it.