search

lippertmarkus / helm-charts

11 stars 7 forks source link

Stuck at "Starting Business Central Tier" #5

Open eokorieFA opened 3 years ago

eokorieFA commented 3 years ago

Hi,

Hope you can help. I am currently attempting to deploy Business Central to Kubernetes cluster using Helm and Terraform. I am sure I have configured the helm file correctly as checking the logs I can see the Helm setting up the application. However, when it gets to "Starting Business Central Tier" , the application just stays stuck there and the installation process being again. I have a timeout of 15 mins in place to make sure that the intall process does not run unncessarily for too long.

Below is a log from the pod:

Adding BUSINESS-CENTRA to hosts file
Using artifactUrl https://bcartifacts.azureedge.net/onprem/16.5.15897.15953/w1
Downloading application artifact /onprem/16.5.15897.15953/w1
Downloading C:\Users\ContainerAdministrator\AppData\Local\Temp\4cdca395-c301-4ba2-a40c-369c3c8ed8ec.zip
Unpacking application artifact
Downloading platform artifact /onprem/16.5.15897.15953/platform
Downloading C:\Users\ContainerAdministrator\AppData\Local\Temp\4a4cd2df-0c98-4eca-b346-162334e37017.zip
Unpacking platform artifact
Downloading Prerequisite Components
Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
Starting Internet Information Server
Copying Service Tier Files
c:\dl\onprem\16.5.15897.15953\platform\ServiceTier\Program Files
c:\dl\onprem\16.5.15897.15953\platform\ServiceTier\System64Folder
Copying PowerShell Scripts
c:\dl\onprem\16.5.15897.15953\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
c:\dl\onprem\16.5.15897.15953\platform\WindowsPowerShellScripts\WebSearch
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Determining Database Collation from c:\dl\onprem\16.5.15897.15953\w1\database\Demo Database NAV (16-0).bak
Restoring CRONUS Demo Database
Setting CompatibilityLevel for CRONUS on localhost\SQLEXPRESS
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Copying Web Client Files
c:\dl\onprem\16.5.15897.15953\platform\WebClient\Microsoft Dynamics NAV
Copying Client Files
c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\program files\Microsoft Dynamics NAV
c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\systemFolder
Copying ModernDev Files
c:\dl\onprem\16.5.15897.15953\platform
c:\dl\onprem\16.5.15897.15953\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\onprem\16.5.15897.15953\platform\ConfigurationPackages
Copying Test Assemblies
C:\dl\onprem\16.5.15897.15953\platform\Test Assemblies
Copying Applications
C:\dl\onprem\16.5.15897.15953\platform\Applications
Starting Business Central Service Tier

Terraform Helm release:

resource "kubernetes_namespace" "business_central" {
  metadata {
    name = "business-central"
  }
}

resource "helm_release" "business_central" {
  name          = "business-central"
  repository    = "https://charts.lippertmarkus.com"
  chart         = "business-central"
  namespace     = kubernetes_namespace.business_central.metadata.0.name

  timeout = 900 // 15 mins wait timeout

  values = [file("configurations/business_central/values.yaml")]

//  depends_on = [
//      helm_release.ingress
//  ]
}

This is the yaml file I am using for the deployment of the application

probes:
  enabled: false

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/cluster-issuer: letsencrypt-staging
    kubernetes.io/tls-acme: "true"
  path: /bc/(/|$)(.*)
  hosts:
    - host: bcentral-dev.domain.io
      path: /bc(/|$)(.*)
      serviceport: web
  tls:
    - secretName: bcentral-dev-tls
      hosts:
        - bcentral-dev.domain.io

Any ideas as to what may be going wrong here? The windows node version is at AKSWindows-2019-17763.2213.210922 while the container version is docker://20.10.6

Thanks,

Emmanuel

lippertmarkus commented 3 years ago

Hi Emmanuel,

can you have a look at the pod events what caused the restart? And especially on the liveness, readiness and startup probes? Maybe your timeout is too short.

I'm having a very similar environment:

NAME                          STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
aks-lin-27577304-vmss000000   Ready    agent   19m   v1.20.9   10.1.1.4      <none>        Ubuntu 18.04.6 LTS               5.4.0-1059-azure   containerd://1.4.9+azure
akswin000000                  Ready    agent   13m   v1.20.9   10.1.1.35     <none>        Windows Server 2019 Datacenter   10.0.17763.2213    docker://20.10.6

Pulling the image already takes around 8 minutes for me:

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m22s  default-scheduler  Successfully assigned default/bc1-business-central-5fc96d495-xsx4v to akswin000000
  Normal  Pulling    8m14s  kubelet            Pulling image "mcr.microsoft.com/businesscentral:10.0.17763.1397"
  Normal  Pulled     26s    kubelet            Successfully pulled image "mcr.microsoft.com/businesscentral:10.0.17763.1397" in 7m48.3140144s
  Normal  Created    26s    kubelet            Created container business-central
  Normal  Started    19s    kubelet            Started container business-central

Starting up the pod with installing the business central artifacts on demand takes me another 7 minutes:

2021-10-12T08:46:10.404047300Z Adding BC1-BUSINESS-CE to hosts file
2021-10-12T08:46:10.455997500Z Using artifactUrl https://bcartifacts.azureedge.net/onprem/16.5.15897.15953/w1
2021-10-12T08:46:10.484253400Z Downloading application artifact /onprem/16.5.15897.15953/w1
2021-10-12T08:46:10.503714100Z Downloading C:\Users\ContainerAdministrator\AppData\Local\Temp\049d2b9c-541c-4802-aff2-c09d7ae4a40a.zip
2021-10-12T08:46:16.085104800Z Unpacking application artifact
2021-10-12T08:46:18.886973300Z Downloading platform artifact /onprem/16.5.15897.15953/platform
2021-10-12T08:46:18.888969500Z Downloading C:\Users\ContainerAdministrator\AppData\Local\Temp\1a4a158a-6590-4f1d-a662-1b5ac3d51f96.zip
2021-10-12T08:46:45.502503200Z Unpacking platform artifact
2021-10-12T08:48:06.044515500Z Downloading Prerequisite Components
2021-10-12T08:48:06.066517500Z Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\DotNetCore\DotNetCore.1.0.4_1.1.1-WindowsHosting.exe
2021-10-12T08:48:07.124905100Z Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\Open XML SDK 2.5 for Microsoft Office\OpenXMLSDKv25.msi
2021-10-12T08:48:07.176859400Z Downloading c:\dl\onprem\16.5.15897.15953\platform\Prerequisite Components\IIS URL Rewrite Module\rewrite_2.0_rtw_x64.msi
2021-10-12T08:48:07.283076500Z Using installer from C:\Run\150-new
2021-10-12T08:48:07.450838200Z Installing Business Central
2021-10-12T08:48:07.462837700Z Installing from artifacts
2021-10-12T08:48:07.476920300Z Starting Local SQL Server
2021-10-12T08:48:14.492439900Z Starting Internet Information Server
2021-10-12T08:48:14.908260300Z Copying Service Tier Files
2021-10-12T08:48:14.915665100Z c:\dl\onprem\16.5.15897.15953\platform\ServiceTier\Program Files
2021-10-12T08:48:15.910255500Z c:\dl\onprem\16.5.15897.15953\platform\ServiceTier\System64Folder
2021-10-12T08:48:15.942754600Z Copying PowerShell Scripts
2021-10-12T08:48:15.942754600Z c:\dl\onprem\16.5.15897.15953\platform\WindowsPowerShellScripts\Cloud\NAVAdministration
2021-10-12T08:48:15.981094900Z c:\dl\onprem\16.5.15897.15953\platform\WindowsPowerShellScripts\WebSearch
2021-10-12T08:48:16.953312000Z Copying dependencies
2021-10-12T08:48:17.247368500Z Copying ReportBuilder
2021-10-12T08:48:17.747784400Z Importing PowerShell Modules
2021-10-12T08:48:18.066840500Z Determining Database Collation from c:\dl\onprem\16.5.15897.15953\w1\database\Demo Database NAV (16-0).bak
2021-10-12T08:48:30.870537800Z Restoring CRONUS Demo Database
2021-10-12T08:48:39.673426100Z Setting CompatibilityLevel for CRONUS on localhost\SQLEXPRESS
2021-10-12T08:48:40.712257000Z Modifying Business Central Service Tier Config File for Docker
2021-10-12T08:48:40.750583000Z Creating Business Central Service Tier
2021-10-12T08:48:40.921751400Z Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
2021-10-12T08:48:41.219622700Z Copying Web Client Files
2021-10-12T08:48:41.219622700Z c:\dl\onprem\16.5.15897.15953\platform\WebClient\Microsoft Dynamics NAV
2021-10-12T08:48:41.220622700Z Copying Client Files
2021-10-12T08:48:41.220622700Z c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\program files\Microsoft Dynamics NAV
2021-10-12T08:48:41.220622700Z c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\program files\Microsoft Dynamics NAV
2021-10-12T08:48:41.220622700Z c:\dl\onprem\16.5.15897.15953\platform\LegacyDlls\systemFolder
2021-10-12T08:48:41.221619700Z Copying ModernDev Files
2021-10-12T08:48:41.221619700Z c:\dl\onprem\16.5.15897.15953\platform
2021-10-12T08:48:41.221619700Z c:\dl\onprem\16.5.15897.15953\platform\ModernDev\program files\Microsoft Dynamics NAV
2021-10-12T08:48:41.221619700Z Copying additional files
2021-10-12T08:48:41.222622100Z Copying ConfigurationPackages
2021-10-12T08:48:41.222622100Z C:\dl\onprem\16.5.15897.15953\platform\ConfigurationPackages
2021-10-12T08:48:41.222622100Z Copying Test Assemblies
2021-10-12T08:48:41.222622100Z C:\dl\onprem\16.5.15897.15953\platform\Test Assemblies
2021-10-12T08:48:41.223620700Z Copying Applications
2021-10-12T08:48:41.223620700Z C:\dl\onprem\16.5.15897.15953\platform\Applications
2021-10-12T08:48:41.225621800Z Starting Business Central Service Tier
2021-10-12T08:51:25.759522500Z Importing license file
2021-10-12T08:51:26.444243200Z Stopping Business Central Service Tier
2021-10-12T08:51:26.710169500Z Installation took 199 seconds
2021-10-12T08:51:26.711170600Z Installation complete
2021-10-12T08:51:26.760492200Z Initializing...
2021-10-12T08:51:26.915700400Z Starting Container
2021-10-12T08:51:26.928125300Z Hostname is bc1-business-central-5fc96d495-xsx4v
2021-10-12T08:51:26.928125300Z PublicDnsName is bc1-business-central-5fc96d495-xsx4v
2021-10-12T08:51:27.760318100Z Using NavUserPassword Authentication
2021-10-12T08:51:28.065976300Z Creating Self Signed Certificate
2021-10-12T08:51:29.685779800Z Self Signed Certificate Thumbprint 95EBB0E7AEA348F2A860BB94FC7872E900F3300B
2021-10-12T08:51:29.898254200Z Modifying Service Tier Config File with Instance Specific Settings
2021-10-12T08:51:30.863462000Z Starting Service Tier
2021-10-12T08:52:08.727401200Z Registering event sources
2021-10-12T08:52:08.799445400Z Creating DotNetCore Web Server Instance
2021-10-12T08:52:37.631773300Z Creating http download site
2021-10-12T08:52:38.799983100Z Setting SA Password and enabling SA
2021-10-12T08:52:38.831984900Z Creating admin as SQL User and add to sysadmin
2021-10-12T08:52:39.749769200Z Creating SUPER user
2021-10-12T08:52:50.161008600Z This was created by Kubernetes.
2021-10-12T08:52:51.271887300Z Container IP Address: 10.1.1.36
2021-10-12T08:52:51.271887300Z Container Hostname  : bc1-business-central-5fc96d495-xsx4v
2021-10-12T08:52:51.272887800Z Container Dns Name  : bc1-business-central-5fc96d495-xsx4v
2021-10-12T08:52:51.273889600Z Web Client          : http://bc1-business-central-5fc96d495-xsx4v/BC/
2021-10-12T08:52:51.276161200Z Dev. Server         : http://bc1-business-central-5fc96d495-xsx4v
2021-10-12T08:52:51.276161200Z Dev. ServerInstance : BC
2021-10-12T08:52:51.290137200Z
2021-10-12T08:52:51.290137200Z Files:
2021-10-12T08:52:51.294974500Z http://bc1-business-central-5fc96d495-xsx4v:8080/ALLanguage.vsix
2021-10-12T08:52:51.294974500Z
2021-10-12T08:52:51.295978300Z WARNING: You are running a container which is 324 days old.
2021-10-12T08:52:51.295978300Z Microsoft recommends that you always run the latest version of our containers.
2021-10-12T08:52:51.296977900Z
2021-10-12T08:52:51.359083900Z Container Total Physical Memory is 7.0Gb
2021-10-12T08:52:51.361195800Z Container Free Physical Memory is 4.0Gb
2021-10-12T08:52:51.361195800Z
2021-10-12T08:52:51.362199300Z Initialization took 85 seconds
2021-10-12T08:52:51.362199300Z Ready for connections!
2021-10-12T08:52:51.385286600Z Starting EventLog Monitor
2021-10-12T08:52:51.447584200Z Monitoring EventSources from EventLog[Application]:
2021-10-12T08:52:51.448584200Z - MicrosoftDynamicsNAVClientClientService
2021-10-12T08:52:51.448584200Z - MicrosoftDynamicsNAVClientWebClient
2021-10-12T08:52:51.449583000Z - MicrosoftDynamicsNavServer$BC
2021-10-12T08:52:51.449583000Z - MSSQL$SQLEXPRESS

You could also have a look at the log timestamps with kubectl logs --timestamps=true deployment/yourdeployment. I guess your timeout is too low.

Does it work when you manually deploy it via helm and not through terraform like I described here: https://lippertmarkus.com/2020/10/01/deploy-bc-helm/?

lippertmarkus commented 3 years ago

I just published v1.0.5 of the Helm chart and increased the startupProbe timeout a bit, maybe that already helps you :)

eokorieFA commented 2 years ago

Sorry for the delayed reply. I haven't gotten round to testing this due to workload. But will definitely be trying it again. Thank you :-)