NAV 2016 - Missing Control AddIns / Cannot compile all objects

Describe the issue After using the NAV2016 Docker Containers for several years, today our Build Server fails every build with:

##[error]: [31588355] Could not load type 
'Microsoft.Dynamics.Nav.Client.TimelineVisualization;PublicKeyToken=31bf3856ad364e35'. -- Object: Page 5540 Item 
Availability by Timeline
[23462397] You have specified an unknown variable.
RemoveSpecialDates
Define the variable under 'Global C/AL symbols'. -- Object: Page 5540 Item Availability by Timeline

Our build server compiles all objects, and this seems to have started today. I was able to reproduce, by creating a cronus container and compiling Page 5540 - Item Availability by Timeline.

Scripts used to create container and cause the issue

$containerName = 'nav2016'
$password = '<Password Here>'
$securePassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$credential = New-Object pscredential 'admin', $securePassword
$auth = 'UserPassword'
$artifactUrl = Get-NavArtifactUrl -nav '2016' -cu '32' -country 'na'
$licenseFile = 'c:\AdvaniaGIT\license\License.flf'
New-BcContainer `
    -accept_eula `
    -containerName $containerName `
    -credential $credential `
    -auth $auth `
    -artifactUrl $artifactUrl `
    -imageName 'nav2016cu32' `
    -licenseFile $licenseFile `
    -dns '8.8.8.8' `
    -includeCSIDE -doNotExportObjectsToText `
    -updateHosts `
    -myscripts @("https://raw.githubusercontent.com/microsoft/nav-docker/master/override/SelfSignedCertificateEx/SetupCertificate.ps1")

Full output of scripts

BcContainerHelper version 3.0.0
BcContainerHelper emits usage statistics telemetry to Microsoft
BcContainerHelper is version 3.0.0
BcContainerHelper is running as administrator
Host is Microsoft Windows 11 Pro - 21H2
Docker Client Version is 20.10.12
Docker Server Version is 20.10.12
Removing nav2016 from container hosts file
Removing nav2016-* from container hosts file
Removing C:\ProgramData\BcContainerHelper\Extensions\nav2016
Fetching all docker images
Fetching all docker volumes
ArtifactUrl and ImageName specified
WARNING: Unable to find matching generic image for your host OS. Using mcr.microsoft.com/businesscentral:10.0.20348.469
Building image nav2016cu32:onprem-9.0.50357.0-na based on mcr.microsoft.com/businesscentral:10.0.20348.469 with https://bcartifacts.azureedge.net/onprem/9.0.50357.0/na
Pulling latest image mcr.microsoft.com/businesscentral:10.0.20348.469
10.0.20348.469: Pulling from businesscentral
Digest: sha256:b9c0b87c83ebfd443332d9cfb1e494ea6e043d614f2689e7c44559ddabbe7ece
Status: Image is up to date for mcr.microsoft.com/businesscentral:10.0.20348.469
mcr.microsoft.com/businesscentral:10.0.20348.469
Generic Tag: 1.0.2.1
Container OS Version: 10.0.20348.469 (ltsc2022)
Host OS Version: 10.0.22000.438 (21H2)
Using process isolation
Using license file c:\AdvaniaGIT\license\Cryobank.flf
Files in c:\bcartifacts.cache\o1jnytsr.vm0\my:
Copying Platform Artifacts
c:\bcartifacts.cache\onprem\9.0.50357.0\platform
Copying Database
Copying Licensefile
Copying Installers
C:\bcartifacts.cache\onprem\9.0.50357.0\na\Installers
Copying TestToolKit
C:\bcartifacts.cache\onprem\9.0.50357.0\na\TestToolKit
Copying UpgradeToolKit
C:\bcartifacts.cache\onprem\9.0.50357.0\na\UpgradeToolKit
c:\bcartifacts.cache\o1jnytsr.vm0
Sending build context to Docker daemon    974MB

Step 1/6 : FROM mcr.microsoft.com/businesscentral:10.0.20348.469
 ---> 3aa1342d1dfb
Step 2/6 : ENV DatabaseServer=localhost DatabaseInstance=SQLEXPRESS DatabaseName=CRONUS IsBcSandbox=N artifactUrl=https://bcartifacts.azureedge.net/onprem/9.0.50357.0/na filesOnly=False
 ---> Using cache
 ---> 24a978004ed2
Step 3/6 : COPY my /run/
 ---> Using cache
 ---> 8728f5f8664b
Step 4/6 : COPY NAVDVD /NAVDVD/
 ---> e3aa37e8ac6d
Step 5/6 : RUN \Run\start.ps1 -installOnly
 ---> Running in b1f222a40439
Using installer from C:\Run\90
Installing NAV
Starting Local SQL Server
Installing Url Rewrite
Installing SQL Clr Types
Installing Report Viewer
Installing OpenXML
Starting Internet Information Server
Copying Service Tier Files
Copying Web Client Files
Copying Windows Client Files
Copying PowerShell Scripts
Copying Test Assemblies
Copying TestToolKit
Copying UpgradeToolKit
Copying ClientUserSettings
Copying ReportBuilder
Restoring CRONUS Demo Database
Determining Database Collation
Restoring CRONUS Demo Database
Installing C:\NAVDVD\Installers\CA.en\OlAddin\OutlookAddIn.Local.CA.en.msi
Installing C:\NAVDVD\Installers\CA.en\RTC\RoleTailoredClient.Local.CA.en.msi
Installing C:\NAVDVD\Installers\CA.en\Server\Server.Local.CA.en.msi
Installing C:\NAVDVD\Installers\CA.en\WebClient\WebClient.Local.CA.en.msi
Skipping C:\NAVDVD\Installers\CA.en\WebHelp\WebHelp.Local.CA.en.msi
Installing C:\NAVDVD\Installers\CA.fr\OlAddin\OutlookAddIn.Local.CA.fr.msi
Installing C:\NAVDVD\Installers\CA.fr\RTC\RoleTailoredClient.Local.CA.fr.msi
Installing C:\NAVDVD\Installers\CA.fr\Server\Server.Local.CA.fr.msi
Installing C:\NAVDVD\Installers\CA.fr\WebClient\WebClient.Local.CA.fr.msi
Skipping C:\NAVDVD\Installers\CA.fr\WebHelp\WebHelp.Local.CA.fr.msi
Installing C:\NAVDVD\Installers\MX.es\OlAddin\OutlookAddIn.Local.MX.es.msi
Installing C:\NAVDVD\Installers\MX.es\RTC\RoleTailoredClient.Local.MX.es.msi
Installing C:\NAVDVD\Installers\MX.es\Server\Server.Local.MX.es.msi
Installing C:\NAVDVD\Installers\MX.es\WebClient\WebClient.Local.MX.es.msi
Skipping C:\NAVDVD\Installers\MX.es\WebHelp\WebHelp.Local.MX.es.msi
Installing C:\NAVDVD\Installers\US.en\OlAddin\OutlookAddIn.Local.US.en.msi
Installing C:\NAVDVD\Installers\US.en\RTC\RoleTailoredClient.Local.US.en.msi
Installing C:\NAVDVD\Installers\US.en\Server\Server.Local.US.en.msi
Installing C:\NAVDVD\Installers\US.en\WebClient\WebClient.Local.US.en.msi
Skipping C:\NAVDVD\Installers\US.en\WebHelp\WebHelp.Local.US.en.msi
Modifying NAV Service Tier Config File for Docker
Creating NAV Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting NAV Service Tier
Importing CRONUS license file
Stopping NAV Service Tier
Installation took 99 seconds
Installation complete
Removing intermediate container b1f222a40439
 ---> 19d6a80d0807
Step 6/6 : LABEL legal="http://go.microsoft.com/fwlink/?LinkId=837447"       created="202201220726"       nav="2016"       cu="cu32"       country="na"       version="9.0.50357.0"       platform="9.0.50357.0"
 ---> Running in 813dd1528ec1
Removing intermediate container 813dd1528ec1
 ---> fed4389ebc27
Successfully built fed4389ebc27
Successfully tagged nav2016cu32:onprem-9.0.50357.0-na
Building image took 166 seconds
Using image nav2016cu32:onprem-9.0.50357.0-na
Creating Container nav2016
Style: onprem
Multitenant: No
Version: 9.0.50357.0
Platform: 9.0.50357.0
Generic Tag: 1.0.2.1
Container OS Version: 10.0.20348.469 (ltsc2022)
Host OS Version: 10.0.22000.438 (21H2)
Using process isolation
Using locale en-US
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Downloading C:\ProgramData\BcContainerHelper\Extensions\nav2016\my\SetupCertificate.ps1
Using license file c:\AdvaniaGIT\license\Cryobank.flf
Files in C:\ProgramData\BcContainerHelper\Extensions\nav2016\my:
- AdditionalOutput.ps1
- AdditionalSetup.ps1
- license.flf
- MainLoop.ps1
- SetupCertificate.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container nav2016 from image nav2016cu32:onprem-9.0.50357.0-na
f38300087075b3593b729ef4a1e27c5ae3211c0495d4a3792c18b76fc20cff6d
Waiting for container nav2016 to be ready
Initializing...
Setting host.docker.internal to 192.168.2.240 in container hosts file (copy from host hosts file)
Setting gateway.docker.internal to 192.168.2.240 in container hosts file (copy from host hosts file)
Setting kubernetes.docker.internal to 127.0.0.1 in container hosts file (copy from host hosts file)
Setting host.containerhelper.internal to 172.26.64.1 in container hosts file
Starting Container
Hostname is nav2016
PublicDnsName is nav2016
Using NavUserPassword Authentication
Starting Local SQL Server
Starting Internet Information Server
Creating Self Signed Certificate
Self Signed Certificate Thumbprint 23D7EEC33F8189184585AB3F7CB16E243CA9C8AD
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Creating Web Site
Creating Web Server Instance
Creating ManagementServicesPort and setting it to 7045
Using license file 'c:\run\my\license.flf'
Import License
Creating http download site
Setting SA Password and enabling SA
Creating admin as SQL User and add to sysadmin
Creating SUPER user
Container IP Address: 172.26.69.236
Container Hostname  : nav2016
Container Dns Name  : nav2016
Web Client          : http://nav2016/NAV/WebClient/
Setting nav2016 to 172.26.69.236 in host hosts file

Files:

Container Total Physical Memory is 31.8Gb
Container Free Physical Memory is 15.8Gb

Initialization took 47 seconds
Ready for connections!
Reading CustomSettings.config from nav2016
Creating Desktop Shortcuts for nav2016
Container nav2016 successfully created

Use:
Get-BcContainerEventLog -containerName nav2016 to retrieve a snapshot of the event log from the container
Get-BcContainerDebugInfo -containerName nav2016 to get debug information about the container
Enter-BcContainer -containerName nav2016 to open a PowerShell prompt inside the container
Remove-BcContainer -containerName nav2016 to remove the container again
docker logs nav2016 to retrieve information about URL's again
...

Screenshots The Client Add-in APPEAR to be there, however compiling, you get this:

Running the page:

Additional context

does it happen all the time? Yes
did it use to work? Yes - Until today or yesterday

I wonder if your changes relating to the self-signed certificate and the generic images has something to do with it?

OS: Windows Server 2019 1809 build 17763.2366

microsoft / navcontainerhelper

NAV 2016 - Missing Control AddIns / Cannot compile all objects #2278