NextMajor does not become healthy

[NAVRockClimber]

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue If I create a nextmajor container it does not become healthy which is critical if you are e.g. in docker swarm or kubernetes environment behind a reverse proxy. I checked the event log and found the error below.

Testing: #2907

The description for Event ID 0 from source MicrosoftDynamicsNAVClientWebClient cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

Category: Microsoft.Dynamics.Nav.Client.Web.ThrottledSystemHealthChecker
EventId: 0
ConnectionId: 0HMOF48DCV641
RequestId: 0HMOF48DCV641:00000005
RequestPath: /BC/Health/System
ActionId: 4d6a27ef-bf7b-41b8-b2c6-256dce78d5fa
ActionName: Microsoft.Dynamics.Nav.WebClient.Controllers.HealthController.System (Prod.Client.WebCoreApp)
MemberName: CheckHealthAndUpdateState
FilePath: s:\repo\src\Platform\Client\Web\Prod.Client.WebCommon\Services\SystemHealth\ThrottledSystemHealthChecker.cs
LineNumber: 150

Exception: 
System.InvalidOperationException: Performance counters are not initialized
   at Microsoft.Dynamics.Nav.Common.PerformanceCounterManager.GetCounterValue(ClientPerformanceCounter counter) in s:\repo\src\Platform\ClientServerShared\Prod.Common\PerformanceCounters\PerformanceCounterManager.cs:line 173
   at Microsoft.Dynamics.Nav.Common.PerformanceCounterReader.GetNstConnectionRatio() in s:\repo\src\Platform\ClientServerShared\Prod.Common\PerformanceCounters\PerformanceCounterReader.cs:line 54
   at Microsoft.Dynamics.Nav.Client.Web.PerformanceSystemHealthChecker.CheckHealth(String stateId, Func`1 healthCheck) in s:\repo\src\Platform\Client\Web\Prod.Client.WebCommon\Services\SystemHealth\PerformanceSystemHealthChecker.cs:line 70
   at Microsoft.Dynamics.Nav.Client.Web.PerformanceSystemHealthChecker.IsSystemHealthyAsync() in s:\repo\src\Platform\Client\Web\Prod.Client.WebCommon\Services\SystemHealth\PerformanceSystemHealthChecker.cs:line 45
   at Microsoft.Dynamics.Nav.Client.Web.ThrottledSystemHealthChecker.CheckHealthAndUpdateState(String stateId, Func`1 healthCheck) in s:\repo\src\Platform\Client\Web\Prod.Client.WebCommon\Services\SystemHealth\ThrottledSystemHealthChecker.cs:line 146

Element not found

Scripts used to create container and cause the issue

$artifacturl=Get-BCArtifactUrl -country de -type Sandbox -select NextMajor -sasToken $sastoken
New-BCImage -includeTestToolkit -includeTestLibrariesOnly -includeTestFrameworkOnly -memory 10G -artifactUrl $ArtifactURl -imageName "<XXX>.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de" -myscripts @("https://raw.githubusercontent.com/microsoft/nav-docker/master/generic/Run/210-new/navinstall.ps1")

New-BcContainer -accept_eula -containerName Test -imageName <XXX>.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de -includeAL -Credential $credential -doNotExportObjectsToText -licenseFile C:\Temp\dbclic-bc21-de.bclicense

Full output of scripts

New-BCImage:

WARNING: Unable to find matching generic image for your host OS. Using mcr.microsoft.com/businesscentral:10.0.20348.1547
cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de
Building image cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de based on mcr.microsoft.com/businesscentral:10.0.20348.1547 with https://bcinsider.azureedge.net/sandbox/22.0.53239.0/de
Pulling latest image mcr.microsoft.com/businesscentral:10.0.20348.1547
10.0.20348.1547: Pulling from businesscentral
Digest: sha256:6ac11be13708ef8c49fb71ce321db8987d843e371a202662bf608d4593d38774
Status: Image is up to date for mcr.microsoft.com/businesscentral:10.0.20348.1547
mcr.microsoft.com/businesscentral:10.0.20348.1547
Generic Tag: 1.0.2.14
Container OS Version: 10.0.20348.1547 (ltsc2022)
Host OS Version: 10.0.22000.1455 (21H2)
Using process isolation
Downloading c:\bcartifacts.cache\13w2322k.0cu\my\navinstall.ps1
Files in c:\bcartifacts.cache\13w2322k.0cu\my:
- navinstall.ps1
Copying Platform Artifacts
c:\bcartifacts.cache\sandbox\22.0.53239.0\platform
Copying Database
Copying Licensefile
Copying ConfigurationPackages
C:\bcartifacts.cache\sandbox\22.0.53239.0\de\ConfigurationPackages
Copying Extensions
C:\bcartifacts.cache\sandbox\22.0.53239.0\de\Extensions
Copying Applications.DE
C:\bcartifacts.cache\sandbox\22.0.53239.0\de\Applications.DE
Untagged: cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de
c:\bcartifacts.cache\13w2322k.0cu
Cannot include TestToolkit without a licensefile, please specify licensefile
Sending build context to Docker daemon  2.449GB

Step 1/6 : FROM mcr.microsoft.com/businesscentral:10.0.20348.1547
 ---> 3fe440a53f97
Step 2/6 : ENV DatabaseServer=localhost DatabaseInstance=SQLEXPRESS DatabaseName=CRONUS IsBcSandbox=Y artifactUrl=https://bcinsider.azureedge.net/sandbox/22.0.53239.0/de?sv=2021-06-08&ss=b&srt=sco&spr=https&st=2022-09-15T00%3A00%3A00Z&se=2023-04-01T00%3A00%3A00Z&sp=rl&sig=rBQiC8y0zbuWC66i2BJcVhydR3XD2aaz47aJ6q6zEss%3D filesOnly=False
 ---> Running in 15d19414d39b
Removing intermediate container 15d19414d39b
 ---> 7b5df79ecfb3
Step 3/6 : COPY my /run/
 ---> c4a368e23fda
Step 4/6 : COPY NAVDVD /NAVDVD/
 ---> 0b9b9dcf434c
Step 5/6 : RUN \Run\start.ps1 -installOnly -multitenant -includeTestToolkit -includeTestLibrariesOnly
 ---> Running in 0841acc007b6
c:\run\my folder doesn't exist, creating it
Using installer from C:\Run\210-new
navinstall was overridden
Installing Business Central
Installing from DVD
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
C:\NAVDVD\ServiceTier\Program Files
C:\NAVDVD\ServiceTier\System64Folder
Copying PowerShell Scripts
C:\NAVDVD\WindowsPowerShellScripts\Cloud\NAVAdministration
C:\NAVDVD\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
C:\NAVDVD\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
C:\NAVDVD
C:\NAVDVD\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\NAVDVD\ConfigurationPackages
Copying Test Assemblies
C:\NAVDVD\Test Assemblies
Copying Extensions
C:\NAVDVD\Extensions
Copying Applications
C:\NAVDVD\Applications
Copying Applications.DE
C:\NAVDVD\Applications.DE
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing CRONUS license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
WARNING: This license is not compatible with this version of Business Central.
Any is already published
Synchronizing Any
Installing Any
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Library Assert is already published
Synchronizing Library Assert
Installing Library Assert
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Permissions Mock is already published
Synchronizing Permissions Mock
Installing Permissions Mock
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Library Variable Storage is already published
Synchronizing Library Variable Storage
Installing Library Variable Storage
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Test Runner is already published
Synchronizing Test Runner
Installing Test Runner
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Publishing C:\Applications.DE\Microsoft_System Application Test Library_22.0.53239.0.app
WARNING: This license is not compatible with this version of Business Central.
Synchronizing System Application Test Library
Installing System Application Test Library
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Publishing C:\Applications.DE\Microsoft_Tests-TestLibraries_22.0.53239.0.app
WARNING: This license is not compatible with this version of Business Central.
Synchronizing Tests-TestLibraries
Installing Tests-TestLibraries
WARNING: This license is not compatible with this version of Business Central.
Stopping Business Central Service Tier
Installation took 415 seconds
Installation complete
Removing intermediate container 0841acc007b6
 ---> 510b2a35ec27
Step 6/6 : LABEL legal="http://go.microsoft.com/fwlink/?LinkId=837447"       created="202302150705"       nav=""       cu=""       multitenant="Y" country="DE"       version="22.0.53239.0"       platform="22.0.53186.0"
 ---> Running in 228fc8981ecb
Removing intermediate container 228fc8981ecb
 ---> b89081fa3c0d
Successfully built b89081fa3c0d
Successfully tagged cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de
Building image took 654 seconds

New-BcContainer:

BcContainerHelper is version 4.0.6
BcContainerHelper is not running as administrator
UsePsSession is False
Host is Microsoft Windows 11 Enterprise - 21H2
Docker Client Version is 23.0.0
Docker Server Version is 23.0.0
Removing Session Test
Removing container Test
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\Test
Fetching all docker images
Fetching all docker volumes
Using image cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de
Creating Container Test
Style: sandbox
Multitenant: No
Version: 22.0.53239.0
Platform: 22.0.53186.0
Generic Tag: 1.0.2.14
Container OS Version: 10.0.20348.1547 (ltsc2022)
Host OS Version: 10.0.22000.1455 (21H2)
Using process isolation
Using locale de-DE
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\Temp\dbclic-bc21-de.bclicense
Files in C:\ProgramData\BcContainerHelper\Extensions\Test\my:
- AdditionalOutput.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatecontainerhosts.ps1
Creating container Test from image cosmoppi.azurecr.io/bcinsider/cosmo-bc:sandbox-22.0.52926.0-de
04baf501a4b757a6aa3c75cac293ab6f5ebcea93c0439215e0cdd2ed13a0d952
Waiting for container Test to be ready
Initializing...
Setting host.containerhelper.internal to 172.21.224.1 in container hosts file
Starting Container
Hostname is Test
PublicDnsName is Test
Using Windows Authentication
Starting Local SQL Server
Starting Internet Information Server
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Using license file 'c:\run\my\license.bclicense'
Import License
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Creating Windows user bkraetge7094
Setting SA Password and enabling SA
WARNING: This license is not compatible with this version of Business Central.
Creating SUPER user
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Container IP Address: 172.21.231.157
Container Hostname  : Test
Container Dns Name  : Test
Web Client          : http://Test/BC/?tenant=default
Dev. Server         : http://Test
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting Test-default to 127.0.0.1 in container hosts file

Files:
http://Test:8080/ALLanguage.vsix

Container Total Physical Memory is 31.8Gb
Container Free Physical Memory is 15.1Gb

Initialization took 89 seconds
Ready for connections!
Reading CustomSettings.config from Test
Creating Desktop Shortcuts for Test
Creating .net Assembly Reference Folder for VS Code
Copying DLLs from C:\Windows\assembly to assemblyProbingPath
Copying DLLs from C:\Windows\Microsoft.NET\assembly to assemblyProbingPath
Copying DLLs from C:\Program Files\Microsoft Dynamics NAV\220\Service to assemblyProbingPath
Copying DLLs from C:\Test Assemblies\Mock Assemblies to assemblyProbingPath
Container Test successfully created
Health check returns False, restarting container
Removing Session Test
Test
Waiting for container Test to be ready

Initializing...
Setting host.containerhelper.internal to 172.21.224.1 in container hosts file
Restarting Container
PublicDnsName unchanged
Hostname is Test
PublicDnsName is Test
Using Windows Authentication
Starting Local SQL Server
Starting Internet Information Server
Starting Service Tier
Container IP Address: 172.21.230.234
Container Hostname  : Test
Container Dns Name  : Test
Web Client          : http://Test/BC/?tenant=default
Dev. Server         : http://Test
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting Test-default to 127.0.0.1 in container hosts file

Files:
http://Test:8080/ALLanguage.vsix

Container Total Physical Memory is 31.8Gb
Container Free Physical Memory is 15.8Gb

Initialization took 12 seconds
Ready for connections!

Use:
Get-BcContainerEventLog -containerName Test to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName Test to get debug information about the container
Enter-BcContainer -containerName Test to open a PowerShell prompt inside the container
Remove-BcContainer -containerName Test to remove the container again
docker logs Test to retrieve information about URL's again

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

• does it happen all the time? Yes
• did it use to work? Yes

[freddydk]

So, you are saying that the container works - but it registers as unhealthy? If you use -doNotCheckHealth - does it then work?

[NAVRockClimber]

Hi Freddy, yes it seems to be just the healthcheck that fails. At least I can reach the webclient now if I disable the healthcheck like you proposed. I did not test more, yet.

Unfortunately the healthcheck is also critical for a tooling like ours at Cosmo.

If I manually invoke the health endpoint in the container I get an error message:

 Invoke-WebRequest -Uri "http://Test/BC/Health/System" -UseBasicParsing

Invoke-WebRequest : {"result":false}
At line:1 char:1
+ Invoke-WebRequest -Uri "http://Test/BC/Health/System" -UseBasicParsin ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

[freddydk]

Looks like a bug in next major. Will file a bug on the client team

[freddydk]

They accepted the bug, should be fixed within a few days.

[NAVRockClimber]

Thanks, Freddy.