search

microsoft / navcontainerhelper

Official Microsoft repository for BcContainerHelper, a PowerShell module, which makes it easier to work with Business Central Containers on Docker.
MIT License
379 stars 243 forks source link

New-BCContainer fails if container exists before #1006

Closed Update-Mark closed 4 years ago

Update-Mark commented 4 years ago

Describe the issue If attempting to refresh or recreate a container PID 4 has the folder locked open and the process times out.

Scripts used to create container and cause the issue

Scripts work if the container does not exist.

Full output of scripts

NavContainerHelper is version 0.6.5.11
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 19.03.4
Docker Server Version is 19.03.4
Pulling image mcr.microsoft.com/businesscentral/onprem:15.1.37881.38071-se-ltsc2019
15.1.37881.38071-se-ltsc2019: Pulling from businesscentral/onprem
Digest: sha256:2a920ed216ca1b1ade50254d41b9fab54893b5fa80ab98de7f32972692ddd611
Using image mcr.microsoft.com/businesscentral/onprem:15.1.37881.38071-se-ltsc2019
Removing container BC15-CU1-TPP
Removing BC15-CU1-TPP from host hosts file
Removing C:\ProgramData\NavContainerHelper\Extensions\BC15-CU1-TPP
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC15-CU1-TPP (attempts: 1)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC15-CU1-TPP
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC15-CU1-TPP (attempts: 2)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
...

Additional context

Update-Mark commented 4 years ago

This is very similar to #290 @freddydk - if you can recall, when you reproduced this, was it a portainer problem? We are running portainer on that server, but stopping it does not seem to resolve the issue. (although I have not attempted to restart the machine and retry.)

freddydk commented 4 years ago

The fix for 290 was to retry after some time. The reason why it cannot remove the folder is that some file is open (or a windows explorer, a cmd prompt or like)

Update-Mark commented 4 years ago

hm. Something is keeping the folder open, and when I used process explorer to see what could be causing the filelock, it returns PID4 - which is normally the kernel. Even force closing explorer.exe does not clear the lock. (and obviously nothing else is currently open).

freddydk commented 4 years ago

Could you share the script you are using to create the container?

Update-Mark commented 4 years ago

ok...pastebin link: https://pastebin.com/tSNTYTtf

freddydk commented 4 years ago

Nothing special... strange

If you try to remove the folder C:\ProgramData\NavContainerHelper\Extensions\BC15-CU1-TPP - in Windows Explorer, that probably also fails. Have you tried to remove all files in the folder manually to which files are locked (or whether it is the folder) Has this happened always - or is it new? Have you tried a reboot to see whether that changes anything? Have you tried with another name (twice obviously)?

Update-Mark commented 4 years ago

yep manually deleting it fails as well...but all the files do get deleted. I don't know when this started. I know I have had issues with it before, but I know that it used to work sometimes. And I honestly cannot remember if it "Really" stopped working after the portainer install. Rebooting the server clears the lock and I can use the same container name without issue and it does not matter what name I use.

freddydk commented 4 years ago

Did you try to stop the portainer container to see whether that one has anything to do with it?

Update-Mark commented 4 years ago

Yep. Stopping it does not seem to clear the lock. Granted I have not restarted the server since then... i will try that tonight.

freddydk commented 4 years ago

What about restarting the docker service? I am fishing in the blind - I don't think this is related to containerhelper.

Update-Mark commented 4 years ago

yeah same here. But no, restarting the docker service does not unlock the folder either. And I agree, it does not sound like its related...I was grasping at straws myself.
What is strange though is that its the NAV container helper folder...in C:\ProgramData\NavContainerHelper\Extensions\\my Is there some sort of symbolic link here that is keeping the folder open? Anything special about that folder?

Update-Mark commented 4 years ago

image

Different container, but same issue.

freddydk commented 4 years ago

what does

docker ps -a

return?

Update-Mark commented 4 years ago

oh the container DOES get removed - its just the C:\ProgramData\NavContainerHelper\Extensions\my that does not and that kills the entire process:

Update-Mark commented 4 years ago

image

freddydk commented 4 years ago

ok, the my folder is shared as c:\run\my inside the folder, strange that the container is gone and the volume is still locked. What update is on the host ?

freddydk commented 4 years ago

You could try to stop the container before new-navcontainer.

Update-Mark commented 4 years ago

sorry - what do you mean by update? host windows version: image docker and nav container helper versions: image

freddydk commented 4 years ago

Your host hasn't been updated since October 2019 - but I assume that -useBestContainerOS uses a generic with version 17763.805 and goes process isolation? Could you paste the full container generation output? (and mark it with Insert Code in the toolbar)

Update-Mark commented 4 years ago

yes to the process isolation, and I will update the host, I did not realize that the last time it attempted to update the updates failed. I will look into that.

here is the first run (successful)

NavContainerHelper is version 0.6.5.11
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 19.03.4
Docker Server Version is 19.03.4
Pulling image mcr.microsoft.com/businesscentral/onprem:16.1.12629.12805-se-ltsc2019
16.1.12629.12805-se-ltsc2019: Pulling from businesscentral/onprem
Using image mcr.microsoft.com/businesscentral/onprem:16.1.12629.12805-se-ltsc2019
Creating Container BC16-CU1-Test
Version: 16.1.12629.12805-se
Style: onprem
Platform: 16.0.12630.12758
Generic Tag: 0.0.9.99
Container OS Version: 10.0.17763.973 (ltsc2019)
Host OS Version: 10.0.17763.805 (ltsc2019)
A better Generic Container OS exists for your host (mcr.microsoft.com/dynamicsnav:10.0.17763.864-generic)
Creating temp container from mcr.microsoft.com/businesscentral/onprem:16.1.12629.12805-se-ltsc2019 and extract necessary files
Extracting Service Tier and WebClient Files
Extracting Windows Client Files
Extracting Configuration packages
Extracting Test Assemblies
Extracting Test Toolkit
Extracting Upgrade Toolkit
Extracting Extensions
Extracting Applications
Extracting Applications.se
Extracting Files from Run folder
Extracting Database Files
Downloading prerequisites
Downloading C:\ProgramData\NavContainerHelper\16.1.12629.12805-se-Files\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Downloading C:\ProgramData\NavContainerHelper\16.1.12629.12805-se-Files\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
Downloading C:\ProgramData\NavContainerHelper\16.1.12629.12805-se-Files\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Performing cleanup
Removing temp container
Using generic image mcr.microsoft.com/dynamicsnav:10.0.17763.864-generic
Pulling image mcr.microsoft.com/dynamicsnav:10.0.17763.864-generic
10.0.17763.864-generic: Pulling from dynamicsnav
<snip - a bunch of download and verification/>
Digest: sha256:83ebabcbf00b43b8601ea4d94b7996002e050949e458730c50fb76c9344868a2
Status: Downloaded newer image for mcr.microsoft.com/dynamicsnav:10.0.17763.864-generic
Generic Container OS Version: 10.0.17763.864 (ltsc2019)
Generic Tag of better generic: 0.0.9.103
WARNING: The container operating system matches the host operating system, but the revision is different.
If you encounter issues, you might want to specify -isolation hyperv
Using locale sv-SE
Using process isolation
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file \\updfile2\affsys\Utveckling\Licenser\docker_latest.flf
Files in C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test\my:
- AdditionalOutput.ps1
- license.flf
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container BC16-CU1-Test from image mcr.microsoft.com/dynamicsnav:10.0.17763.864-generic
9cd88ff0b84456888725d6e411566232bec49625e25a889de4805c86234f224d
Waiting for container BC16-CU1-Test to be ready
Installing Business Central
Installing Url Rewrite
Installing OpenXML
Installing DotNetCore
Starting Local SQL Server
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
Starting Internet Information Server
Copying Service Tier Files
Copying Web Client Files
Copying Client Files
Copying ModernDev Files
Copying PowerShell Scripts
Copying Test Assemblies
Copying Applications
Copying ReportBuilder
Changing Database Server Collation to Finnish_Swedish_100_CI_AS
Copying Cronus database
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Installation took 727 seconds
Installation complete
Initializing...
Setting host.containerhelper.internal to 172.18.1.1 in container hosts file
Starting Container
Hostname is BC16-CU1-Test
PublicDnsName is BC16-CU1-Test
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint B572B4D6887DA93384D48E6FED229C9FD639FD44
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using license file 'c:\run\my\license.flf'
Import License
Creating http download site
Setting SA Password and enabling SA
Creating updlocal as SQL User and add to sysadmin
Creating SUPER user
Container IP Address: 172.18.2.42
Container Hostname  : BC16-CU1-Test
Container Dns Name  : BC16-CU1-Test
Web Client          : http://BC16-CU1-Test/BC/
Dev. Server         : http://BC16-CU1-Test
Dev. ServerInstance : BC
Setting BC16-CU1-Test to 172.18.2.42 in host hosts file

Files:
http://BC16-CU1-Test:8080/ALLanguage.vsix

Initialization took 1073 seconds
Ready for connections!
Reading CustomSettings.config from BC16-CU1-Test
Creating Desktop Shortcuts for BC16-CU1-Test
Creating .net Assembly Reference Folder for VS Code
Copying DLLs from C:\Windows\assembly to assemblyProbingPath
Copying DLLs from C:\Program Files\Microsoft Dynamics NAV\160\Service to assemblyProbingPath
Copying DLLs from C:\Program Files (x86)\Microsoft Dynamics NAV\160\RoleTailored Client to assemblyProbingPath
Copying DLLs from C:\Test Assemblies\Mock Assemblies to assemblyProbingPath
Copying DLLs from C:\Program Files (x86)\Open XML SDK to assemblyProbingPath
Container BC16-CU1-Test successfully created

and with no changes, I stopped the container and immediately ran the script again.

NavContainerHelper is version 0.6.5.11
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 19.03.4
Docker Server Version is 19.03.4
Pulling image mcr.microsoft.com/businesscentral/onprem:16.1.12629.12805-se-ltsc2019
16.1.12629.12805-se-ltsc2019: Pulling from businesscentral/onprem
Digest: sha256:c12ddcb607b1f07399595280f9f22c308cd83fe9048c5d9d1b6d4980412d161e
Using image mcr.microsoft.com/businesscentral/onprem:16.1.12629.12805-se-ltsc2019
Removing container BC16-CU1-Test
Removing BC16-CU1-Test from host hosts file
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 1)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 2)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 3)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 4)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 5)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 6)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 7)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 8)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 9)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Error removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test (attempts: 10)
Please close any apps, prompts or files using this folder
Retrying in 10 seconds
Removing C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
Could not remove C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
At C:\Program Files\WindowsPowerShell\Modules\navcontainerhelper\0.6.5.11\ContainerHandling\Remove-NavContainer.ps1:64 char:21
+                     throw "Could not remove $containerFolder"
+                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Could not remov...s\BC16-CU1-Test:String) [], RuntimeException
    + FullyQualifiedErrorId : Could not remove C:\ProgramData\NavContainerHelper\Extensions\BC16-CU1-Test
jwinberg commented 4 years ago

I think I'm having this same problem as well now suddenly. Our Devops builds are now all failing at "Remove Build Container" step. I'm not sure yet when this has started happening since some of the build are set up to "reuseContainer", hence not affected with this problem. The VM running agents was created just 2 weeks ago, but version is 1809. Not sure if this is the problem but I will test with a newer windows server version and see if it has any effect next week.

freddydk commented 4 years ago

@jwinberg - what version is the host running? and can you paste an output from the container generation?

jwinberg commented 4 years ago

here is the log from Create Build Container:


Create demoapp-build from mcr.microsoft.com/businesscentral/sandbox:fi
NavContainerHelper is version 0.6.5.11
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 19.03.5
Docker Server Version is 19.03.5
Using image mcr.microsoft.com/businesscentral/sandbox:fi-ltsc2019
Disabling Health Check (always report healthy)
Creating Container demoapp-build
Version: 16.1.12629.12805-fi
Style: sandbox
Platform: 16.0.12630.12758
Generic Tag: 0.0.9.99
Container OS Version: 10.0.17763.973 (ltsc2019)
Host OS Version: 10.0.17763.1217 (ltsc2019)
A better Generic Container OS exists for your host (mcr.microsoft.com/dynamicsnav:10.0.17763.1217-generic)
Using generic image mcr.microsoft.com/dynamicsnav:10.0.17763.1217-generic
Generic Container OS Version: 10.0.17763.1217 (ltsc2019)
Generic Tag of better generic: 0.0.9.102
Using locale fi-FI
Using process isolation
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\Licenses\BC16_DEV_ISV.flf
Files in C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\my:
- AdditionalOutput.ps1
- license.flf
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container demoapp-build from image mcr.microsoft.com/dynamicsnav:10.0.17763.1217-generic
c71dec5ff80b6919afa75bd5c2b4d01db2c75dd1ccc5c4e1f83162df2413edd6
Waiting for container demoapp-build to be ready
Installing Business Central
Installing Url Rewrite
Installing OpenXML
Installing DotNetCore
Starting Local SQL Server
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
Starting Internet Information Server
Copying Service Tier Files
Copying Web Client Files
Copying Client Files
Copying ModernDev Files
Copying PowerShell Scripts
Copying ConfigurationPackages
Copying Test Assemblies
Copying Extensions
Copying Applications
Copying Applications.*
Copying ReportBuilder
Changing Database Server Collation to Finnish_Swedish_100_CS_AS
Copying Cronus database
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Installation took 417 seconds
Installation complete
Stopping Business Central Service Tier
Initializing...
Setting host.containerhelper.internal to 172.22.80.1 in container hosts file
Starting Container
Hostname is demoapp-build
PublicDnsName is demoapp-build
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint 3F75D865633805F805235BBB161A42C5685B4831
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Enabling Financials User Experience
Using license file 'c:\run\my\license.flf'
Import License
Creating http download site
Setting SA Password and enabling SA
Creating admin as SQL User and add to sysadmin
Creating SUPER user
Container IP Address: 172.22.82.80
Container Hostname  : demoapp-build
Container Dns Name  : demoapp-build
Web Client          : http://demoapp-build/BC/
Dev. Server         : http://demoapp-build
Dev. ServerInstance : BC
Setting demoapp-build to 172.22.82.80 in host hosts file

Files:
http://demoapp-build:8080/ALLanguage.vsix

Initialization took 110 seconds
Ready for connections!
Reading CustomSettings.config from demoapp-build
Copy from container demoapp-build (C:\Applications.fi\Microsoft_Any_16.1.12629.12805.app) to C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Any_16.1.12629.12805.app
Publishing C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Any_16.1.12629.12805.app
Synchronizing Any on tenant default
Installing Any on tenant default
App successfully published
Copy from container demoapp-build (C:\Applications.fi\Microsoft_Library Assert_16.1.12629.12805.app) to C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Library Assert_16.1.12629.12805.app
Publishing C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Library Assert_16.1.12629.12805.app
Synchronizing Library Assert on tenant default
Installing Library Assert on tenant default
App successfully published
Copy from container demoapp-build (C:\Applications\TestFramework\TestRunner\Microsoft_Test Runner.app) to C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Test Runner.app
Publishing C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Test Runner.app
Synchronizing Test Runner on tenant default
Installing Test Runner on tenant default
App successfully published
Copy from container demoapp-build (C:\Applications.fi\Microsoft_System Application Test Library_16.1.12629.12805.app) to C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_System Application Test Library_16.1.12629.12805.app
Publishing C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_System Application Test Library_16.1.12629.12805.app
Synchronizing System Application Test Library on tenant default
Installing System Application Test Library on tenant default
App successfully published
Copy from container demoapp-build (C:\Applications.fi\Microsoft_Tests-TestLibraries_16.1.12629.12805.app) to C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Tests-TestLibraries_16.1.12629.12805.app
Publishing C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\_Microsoft_Tests-TestLibraries_16.1.12629.12805.app
Synchronizing Tests-TestLibraries on tenant default
Installing Tests-TestLibraries on tenant default
App successfully published
TestToolkit successfully imported
Container demoapp-build successfully created
Backing up CRONUS to C:\ProgramData\NavContainerHelper\onprem-16.1.12629.12805-fi-bakFolders\demoapp-build\database.bak
##[section]Finishing: Create Build Container

On the server there is a empty folder that cannot be removed C:\ProgramData\NavContainerHelper\Extensions\demoapp-build\my

jwinberg commented 4 years ago

With new navcontainerhelper version 0.7.0.0 this issue is now fixed for us. Or the \my folder is not removed and cannot be removed even manually before a reboot. But at least our build pipelines are not failing on Remove Build Container step. Thanks @freddydk

freddydk commented 4 years ago

Thanks. I changed the testing and start by removing all the files, If any files are locked - I will write out which file it is and wait 10 seconds - this is a hard stop. If all files can be deleted - I will ignore that the folder cannot.

freddydk commented 4 years ago

Please re-open if the issue persists after upgrade to containerhelper 0.7.0.0 or later