Traefik not recognizing labels for BC20.1 container

[JeppeBylov]

PLEASE DO NOT INCLUDE ANY PASSWORDS OR TOKENS IN YOUR ISSUE!!!

Describe the issue We are using traefik for exposing containers externally. We recently decided to start the process of upgrading our containers from 18.4 to 20.1. When we create a container for version 20.1, traefik no longer registers the labels and therefore we get a "Bad Gateway" error when browsing the URL. Works fine for version 18.4, scripts for creating containers are identical except version.

Scripts used to create container and cause the issue

$containerName = "KvikTestBC20"
$authenticationEmail = "jby@kvik.com"
$cetificatefile = "C:\Temp\wildcard_kvik_com_2022.pfx"

$bcartifact = Get-BCArtifactUrl -type OnPrem -country "w1" -version "20.1" -select Latest

$params = @{
    "auth"                = 'AAD'
    "containerName"       = $containerName
    "multitenant"         = $false
    "isolation"           = 'process'
    "artifacturl"         = $bcartifact
    "updateHosts"         = $true
    "useTraefik"          = $true
    "myscripts"           = @("C:\Powershell Scripts\SetupCertificate.ps1", $cetificatefile)
    "publicDnsName"       = 'bc.kvik.com'
    "authenticationEMail" = $authenticationEmail
}

New-BCContainer @params `
    -accept_eula `
    -accept_outdated `
    -Credential $Credential `
    -doNotExportObjectsToText `
    -shortcuts CommonDesktopFolder

Full output of scripts 20.1

BcContainerHelper is version 3.0.10
BcContainerHelper is running as administrator
Hyper-V is Disabled
UsePsSession is True
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 20.10.6
Docker Server Version is 20.10.6
Removing Session KvikTestBC20
Removing container KvikTestBC20
Removing entries from hosts
Removing KvikTestBC20 from container hosts file
Removing KvikTestBC20-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\KvikTestBC20
Fetching all docker images
Fetching all docker volumes
Enabling SSL as otherwise all clients will see mixed HTTP / HTTPS request, which will cause problems e.g. on the mobile and modern windows clients
Using image mcr.microsoft.com/businesscentral:10.0.17763.3046
PublicDnsName is bc.kvik.com
Creating Container KvikTestBC20
Style: onprem
Multitenant: No
Version: 20.1.39764.39901
Platform: 20.0.39668.39849
Generic Tag: 1.0.2.10
Container OS Version: 10.0.17763.3046 (ltsc2019)
Host OS Version: 10.0.17763.3046 (ltsc2019)
Using process isolation
Using locale en-US
Adding special CheckHealth.ps1 to enable Traefik support
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Additional Parameters:
-e webserverinstance=KvikTestBC20
-e publicdnsname=bc.kvik.com
-l "traefik.protocol=https"
-l "traefik.web.frontend.rule=PathPrefix:/KvikTestBC20"
-l "traefik.web.port=443"
-l "traefik.soap.frontend.rule=PathPrefix:/KvikTestBC20soap;ReplacePathRegex: ^/KvikTestBC20soap(.*) /BC$1"
-l "traefik.soap.port=7047"
-l "traefik.rest.frontend.rule=PathPrefix:/KvikTestBC20rest;ReplacePathRegex: ^/KvikTestBC20rest(.*) /BC$1"
-l "traefik.rest.port=7048"
-l "traefik.dev.frontend.rule=PathPrefix:/KvikTestBC20dev;ReplacePathRegex: ^/KvikTestBC20dev(.*) /BC$1"
-l "traefik.dev.port=7049"
-l "traefik.snap.frontend.rule=PathPrefix:/KvikTestBC20snap;ReplacePathRegex: ^/KvikTestBC20snap(.*) /BC$1"
-l "traefik.snap.port=7083"
-l "traefik.dl.frontend.rule=PathPrefixStrip:/KvikTestBC20dl"
-l "traefik.dl.port=8080"
-l "traefik.dl.protocol=http"
-l "traefik.enable=true"
-l "traefik.frontend.entryPoints=https"
--env customNavSettings=PublicODataBaseUrl=https://bc.kvik.com/KvikTestBC20rest/odata,PublicSOAPBaseUrl=https://bc.kvik.com/KvikTestBC20soap/ws,PublicWebBaseUrl=https://bc.kvik.com/KvikTestBC20
Files in C:\ProgramData\BcContainerHelper\Extensions\KvikTestBC20\my:
- AdditionalOutput.ps1
- CheckHealth.ps1
- MainLoop.ps1
- SetupCertificate.ps1
- SetupVariables.ps1
- updatehosts.ps1
- wildcard_kvik_com_2022.pfx
Creating container KvikTestBC20 from image mcr.microsoft.com/businesscentral:10.0.17763.3046
b12ed367426f991e829c82e3bad5523cda2c01d8855d37734aeffad569a2e95b
Waiting for container KvikTestBC20 to be ready
Using artifactUrl https://bcartifacts.azureedge.net/onprem/20.1.39764.39901/w1
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\onprem\20.1.39764.39901\platform\ServiceTier\Program Files
c:\dl\onprem\20.1.39764.39901\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\onprem\20.1.39764.39901\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\onprem\20.1.39764.39901\platform\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
c:\dl\onprem\20.1.39764.39901\platform\WebClient\Microsoft Dynamics NAV
Copying Client Files
c:\dl\onprem\20.1.39764.39901\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\20.1.39764.39901\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\20.1.39764.39901\platform\LegacyDlls\systemFolder
Copying ModernDev Files
c:\dl\onprem\20.1.39764.39901\platform
c:\dl\onprem\20.1.39764.39901\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\onprem\20.1.39764.39901\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\onprem\20.1.39764.39901\platform\Test Assemblies
Copying Applications
C:\dl\onprem\20.1.39764.39901\platform\Applications
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for CRONUS on localhost\SQLEXPRESS
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Stopping Business Central Service Tier
Installation took 101 seconds
Installation complete
Initializing...
Setting host.docker.internal to 10.1.0.21 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 10.1.0.21 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.21.64.1 in container hosts file
Starting Container
Hostname is KvikTestBC20
PublicDnsName is bc.kvik.com
Using AccessControlService Authentication
Certificate File Thumbprint F1FAEFDBEF1AE307D144FB8577549D07B69B2901
Import Certificate to LocalMachine\my
Modifying Service Tier Config File with Instance Specific Settings
Modifying Service Tier Config File with settings from environment variable
Setting PublicODataBaseUrl to https://bc.kvik.com/KvikTestBC20rest/odata
Setting PublicSOAPBaseUrl to https://bc.kvik.com/KvikTestBC20soap/ws
Setting PublicWebBaseUrl to https://bc.kvik.com/KvikTestBC20
Starting Service Tier
CertificateThumprint F1FAEFDBEF1AE307D144FB8577549D07B69B2901
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: KvikTestBC20
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\KvikTestBC20
Create the application pool KvikTestBC20
Create website: NavWebApplicationContainer with SSL
Update configuration: navsettings.json
Done Configuring Web Client
Creating http download site
Setting SA Password and enabling SA
Creating jby@kvik.com as SQL User and add to sysadmin
Creating SUPER user
WARNING: The password that you entered does not meet the minimum requirements. 
It should be at least 8 characters long and contain at least one uppercase 
letter, one lowercase letter, and one number.
Container IP Address: 172.21.73.142
Container Hostname  : KvikTestBC20
Container Dns Name  : bc.kvik.com
Web Client          : https://bc.kvik.com/KvikTestBC20/
Dev. Server         : https://bc.kvik.com
Dev. ServerInstance : BC
Setting KvikTestBC20 to 172.21.73.142 in host hosts file

Files:
http://bc.kvik.com:8080/ALLanguage.vsix

Container Total Physical Memory is 64.0Gb
Container Free Physical Memory is 22.9Gb

Initialization took 66 seconds
Ready for connections!
Reading CustomSettings.config from KvikTestBC20
Creating Desktop Shortcuts for KvikTestBC20
Container KvikTestBC20 successfully created
Because of Traefik, the following URLs need to be used when accessing the container from outside your Docker host:
Web Client:        https://bc.kvik.com/KvikTestBC20
SOAP WebServices:  https://bc.kvik.com/KvikTestBC20soap
OData WebServices: https://bc.kvik.com/KvikTestBC20rest
Dev Service:       https://bc.kvik.com/KvikTestBC20dev
Snapshot Service:  https://bc.kvik.com/KvikTestBC20snap
File downloads:    https://bc.kvik.com/KvikTestBC20dl

Use:
Get-BcContainerEventLog -containerName KvikTestBC20 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName KvikTestBC20 to get debug information about the container
Enter-BcContainer -containerName KvikTestBC20 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName KvikTestBC20 to remove the container again
docker logs KvikTestBC20 to retrieve information about URL's again

18.4

BcContainerHelper is version 3.0.10
BcContainerHelper is running as administrator
Hyper-V is Disabled
UsePsSession is True
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 20.10.6
Docker Server Version is 20.10.6
Removing Session KvikTestBC20
Removing container KvikTestBC20
Removing entries from hosts
Removing KvikTestBC20 from container hosts file
Removing KvikTestBC20-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\KvikTestBC20
Fetching all docker images
Fetching all docker volumes
Enabling SSL as otherwise all clients will see mixed HTTP / HTTPS request, which will cause problems e.g. on the mobile and modern windows clients
Using image mcr.microsoft.com/businesscentral:10.0.17763.3046
PublicDnsName is bc.kvik.com
Creating Container KvikTestBC20
Style: onprem
Multitenant: No
Version: 18.4.28601.29139
Platform: 18.0.28593.29123
Generic Tag: 1.0.2.10
Container OS Version: 10.0.17763.3046 (ltsc2019)
Host OS Version: 10.0.17763.3046 (ltsc2019)
Using process isolation
Using locale en-US
Adding special CheckHealth.ps1 to enable Traefik support
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Additional Parameters:
-e webserverinstance=KvikTestBC20
-e publicdnsname=bc.kvik.com
-l "traefik.protocol=https"
-l "traefik.web.frontend.rule=PathPrefix:/KvikTestBC20"
-l "traefik.web.port=443"
-l "traefik.soap.frontend.rule=PathPrefix:/KvikTestBC20soap;ReplacePathRegex: ^/KvikTestBC20soap(.*) /BC$1"
-l "traefik.soap.port=7047"
-l "traefik.rest.frontend.rule=PathPrefix:/KvikTestBC20rest;ReplacePathRegex: ^/KvikTestBC20rest(.*) /BC$1"
-l "traefik.rest.port=7048"
-l "traefik.dev.frontend.rule=PathPrefix:/KvikTestBC20dev;ReplacePathRegex: ^/KvikTestBC20dev(.*) /BC$1"
-l "traefik.dev.port=7049"
-l "traefik.snap.frontend.rule=PathPrefix:/KvikTestBC20snap;ReplacePathRegex: ^/KvikTestBC20snap(.*) /BC$1"
-l "traefik.snap.port=7083"
-l "traefik.dl.frontend.rule=PathPrefixStrip:/KvikTestBC20dl"
-l "traefik.dl.port=8080"
-l "traefik.dl.protocol=http"
-l "traefik.enable=true"
-l "traefik.frontend.entryPoints=https"
--env customNavSettings=PublicODataBaseUrl=https://bc.kvik.com/KvikTestBC20rest/odata,PublicSOAPBaseUrl=https://bc.kvik.com/KvikTestBC20soap/ws,PublicWebBaseUrl=https://bc.kvik.com/KvikTestBC20
Files in C:\ProgramData\BcContainerHelper\Extensions\KvikTestBC20\my:
- AdditionalOutput.ps1
- CheckHealth.ps1
- MainLoop.ps1
- SetupCertificate.ps1
- SetupVariables.ps1
- updatehosts.ps1
- wildcard_kvik_com_2022.pfx
Creating container KvikTestBC20 from image mcr.microsoft.com/businesscentral:10.0.17763.3046
37f5db20367242332004cb1899a4eab1826576dab5a4e303c05d8575d0c7d569
Waiting for container KvikTestBC20 to be ready
Using artifactUrl https://bcartifacts.azureedge.net/onprem/18.4.28601.29139/w1
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
c:\dl\onprem\18.4.28601.29139\platform\ServiceTier\Program Files
c:\dl\onprem\18.4.28601.29139\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\onprem\18.4.28601.29139\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\onprem\18.4.28601.29139\platform\WindowsPowerShellScripts\WebSearch
Copying Web Client Files
c:\dl\onprem\18.4.28601.29139\platform\WebClient\Microsoft Dynamics NAV
Copying Client Files
c:\dl\onprem\18.4.28601.29139\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\18.4.28601.29139\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\18.4.28601.29139\platform\LegacyDlls\systemFolder
Copying ModernDev Files
c:\dl\onprem\18.4.28601.29139\platform
c:\dl\onprem\18.4.28601.29139\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\onprem\18.4.28601.29139\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\onprem\18.4.28601.29139\platform\Test Assemblies
Copying Applications
C:\dl\onprem\18.4.28601.29139\platform\Applications
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for CRONUS on localhost\SQLEXPRESS
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Stopping Business Central Service Tier
Installation took 91 seconds
Installation complete
Initializing...
Setting host.docker.internal to 10.1.0.21 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 10.1.0.21 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.21.64.1 in container hosts file
Starting Container
Hostname is KvikTestBC20
PublicDnsName is bc.kvik.com
Using AccessControlService Authentication
Certificate File Thumbprint F1FAEFDBEF1AE307D144FB8577549D07B69B2901
Import Certificate to LocalMachine\my
Modifying Service Tier Config File with Instance Specific Settings
Modifying Service Tier Config File with settings from environment variable
Setting PublicODataBaseUrl to https://bc.kvik.com/KvikTestBC20rest/odata
Setting PublicSOAPBaseUrl to https://bc.kvik.com/KvikTestBC20soap/ws
Setting PublicWebBaseUrl to https://bc.kvik.com/KvikTestBC20
Starting Service Tier
CertificateThumprint F1FAEFDBEF1AE307D144FB8577549D07B69B2901
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: KvikTestBC20
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\KvikTestBC20
Create the application pool KvikTestBC20
Create website: NavWebApplicationContainer with SSL
Update configuration: navsettings.json
Done Configuring Web Client
Creating http download site
Setting SA Password and enabling SA
Creating jby@kvik.com as SQL User and add to sysadmin
Creating SUPER user
WARNING: The password that you entered does not meet the minimum requirements. 
It should be at least 8 characters long and contain at least one uppercase 
letter, one lowercase letter, and one number.
Container IP Address: 172.21.77.22
Container Hostname  : KvikTestBC20
Container Dns Name  : bc.kvik.com
Web Client          : https://bc.kvik.com/KvikTestBC20/
Dev. Server         : https://bc.kvik.com
Dev. ServerInstance : BC
Setting KvikTestBC20 to 172.21.77.22 in host hosts file

Files:
http://bc.kvik.com:8080/ALLanguage.vsix

Container Total Physical Memory is 64.0Gb
Container Free Physical Memory is 23.1Gb

Initialization took 60 seconds
Ready for connections!
Reading CustomSettings.config from KvikTestBC20
Creating Desktop Shortcuts for KvikTestBC20
Container KvikTestBC20 successfully created
Because of Traefik, the following URLs need to be used when accessing the container from outside your Docker host:
Web Client:        https://bc.kvik.com/KvikTestBC20
SOAP WebServices:  https://bc.kvik.com/KvikTestBC20soap
OData WebServices: https://bc.kvik.com/KvikTestBC20rest
Dev Service:       https://bc.kvik.com/KvikTestBC20dev
Snapshot Service:  https://bc.kvik.com/KvikTestBC20snap
File downloads:    https://bc.kvik.com/KvikTestBC20dl

Use:
Get-BcContainerEventLog -containerName KvikTestBC20 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName KvikTestBC20 to get debug information about the container
Enter-BcContainer -containerName KvikTestBC20 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName KvikTestBC20 to remove the container again
docker logs KvikTestBC20 to retrieve information about URL's again

Screenshots If applicable, add screenshots to help explain your problem. Picture of 18.4 working - it registers 6 frontends

20.1 - no frontends registered

Additional context

• does it happen all the time?
• did it use to work?

[freddydk]

@tfenster - I don't think this is a general problem that version 20.x doesn't work with traefik - right? Did you try to give the 20.1 container a different name to see whether it is because traefik doesn't see the change?

[tfenster]

No, all good for lots of 20.1 containers.

@JeppeBylov does the container become healthy?

[Bjarke848]

@tfenster I'm working on this project with Jeppe. It seems like the BC 20.1 container becomes unhealthy - I've used the same script as Jeppe with 20.1 and 18.4

[tfenster]

@Bjarke848 then you will need to figure out why the 20 container is unhealthy. Traefik only handles healthy containers

[freddydk]

QQ - do you use the same AAD App Registration for the 18.x and the 20.x instance? There are differences, I cannot remember the details - but they are on docs.

[freddydk]

This PR https://github.com/microsoft/navcontainerhelper/commit/7c29b487263bde96cb99143819bb9db8f91deb2b added OpenIdConnect support for AAD App Registration created by Create-AadAppsForBC (FYI)

[Bjarke848]

@freddydk No, we are not using the same App Registration for 18.4 and 20.1

[freddydk]

I have also created a number of 20.x containers with AAD auth and traefik over the weekend (to test Edit In Excel - another issue) - they work fine for normal usage, so the issue must be local.

Maybe you can use https://aka.ms/getbc to create a VM with AAD auth and traefik and then compare the apps and BC generated with the ones created by the script you use.

[Bjarke848]

@freddydk @tfenster @JeppeBylov I've figured it out. The EventViewer got the exception "AadApplicationId cannot be null or empty".

So instead of running Create-AadAppsForBC after I've created the container, I used an existing AAD App Registration, and added the AadAppId parameter (-AadAppId "**-**...) to the New-BCContainer script.

Script:

$containerName = "KvikTestBC20BVI" $authenticationEmail = "bvi@kvik.com" $cetificatefile = "C:\Temp\wildcard_kvik_com_2022.pfx"

$bcartifact = Get-BCArtifactUrl -type OnPrem -country "w1" -version "20.1" -select Latest

$params = @{ "auth" = 'AAD' "containerName" = $containerName "multitenant" = $false "artifacturl" = $bcartifact "updateHosts" = $true "useTraefik" = $true "myscripts" = @("C:\Powershell Scripts\SetupCertificate.ps1", $cetificatefile) "publicDnsName" = 'bc.kvik.com' "authenticationEMail" = $authenticationEmail }

New-BCContainer @params -accept_eula -accept_outdated -Credential $Credential -doNotExportObjectsToText -shortcuts CommonDesktopFolder -AadAppId "8eba5471-99e5-1111-1111-93fbaf782ec0"

Healthy and possible to reach!