AlopsDockerWait stalls at Starting Local SQL Server

[MortenRa]

I often experience the pipeline hangs af ALOpsDockerWait step Starting Local SQL Server

are there any settings I can add or settings on the Server to configure,

Here is my Yaml:

---

trigger:
  branches:
    include:
    - master
  paths:
    include:
    - app/*
    - test/*

name: $(Build.BuildId)

pool: default

variables:
- group: Secrets
- name: 'AppVersion'
  value: '16.1.[yyyyWW].*'
- name: 'dockerimage'
  value: 'mcr.microsoft.com/businesscentral/sandbox'

steps:
- checkout: self
  clean: true 

- task: ALOpsDockerStart@1
  inputs:
    docker_image: '$(dockerimage)'
    accept_image_outdated: true
    memory_gb: '8'
    container_restart: 'always'
    docker_parameters: '--isolation=hyperv'

- task: ALOpsDockerWait@1
  inputs:
    search_string: 'Ready for connections!'

- task: ALOpsLicenseImport@1
  displayName: 'ALOps License Import'
  inputs:
    usedocker: true
    license_path: $(LicenseFile)

#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

- task: DownloadBuildArtifacts@0
  displayName: 'Download Library App'
  enabled: true
  inputs:
    buildType: 'specific'
    project: 'Abakion Apps'
    pipeline: '1'
    buildVersionToDownload: 'latestFromBranch'
    allowPartiallySucceededBuilds: true
    downloadType: 'specific'
    itemPattern: '**/*.app'

- task: ALOpsAppPublish@1
  displayName: 'Publish Library App'
  enabled: true
  inputs:
    usedocker: true
    nav_artifact_app_filter: '*app.app'
    skip_verification: false

- task: ALOpsAppCompiler@1
  displayName: 'ALOps Compile Extension: App'
  inputs:
    usedocker: true
    targetproject: 'app/app.json'
    nav_app_version: '$(AppVersion)'
    failed_on_warnings: false
    app_file_suffix: 'app'

- task: ALOpsAppSign@1
  displayName: 'ALOps App Sign'
  env:
    pfx_password: $(CodeSignPfxPassword)
  inputs:
    pfx_path: $(CodeSignPfxFile)
    nav_artifact_app_filter: '*app.app'
    timestamp_uri: 'http://timestamp.comodoca.com/authenticode'
    usedocker: True  

- task: ALOpsAppSignVerify@1
  displayName: 'ALOps App Sign Verify'
  inputs:
    usedocker: True
    nav_artifact_app_filter: '*app.app '

- task: ALOpsAppPublish@1
  displayName: 'ALOps Publish Extension: app'
  inputs:
    usedocker: true
    nav_artifact_app_filter: '*app.app'
    skip_verification: false   

- task: ALOpsAppRuntimePackage@1
  inputs:
    usedocker: true
    targetproject: 'app/app.json'
    publish_artifact: true

#<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

- task: ALOpsDockerRemove@1
  displayName: 'ALOps Remove Docker Container'
  enabled: true
  condition: always()

[waldo1001]

Hi Morten,

A few suggestions:

• Are you sure you have enough resources in the server?
• Did you try up the mem to 16Gb (if enough available)?
• Did you try with and without hyper-v isolation?
• Can you remove the "container restart" option?
• Does it happen on other DevOps agents as well? Would you be able to set up one and try?

What is "often"? And does it hang more than an hour so your pipeline fails?

Soon (after the weekend), we'll release a new step "AlopsDockerCreate" that can create your images based on BC Artifacts. This should also help with the stability.

[MortenRa]

Yes

• I should have plenty of resources and even tried with 3 different agents.
• I will try to up the mem to 16gb
• I will try without the hyper-v isolation.
• The Container-restart was added yesterday to see if that helped.
• It happens everyday and it hangs more than an hour so my pipeline fails, then I can restart the failed and it goes through.

[waldo1001]

Would you be able to also try with a build agent that is not in your network? Like "aka.ms/getbuildagent"

[MortenRa]

The build agent is not in my network.

Removing the hyper-v isolation seems to give me another problem in step ALOps Compile Extension: App now it can not find my path. (Same error in 2 different pipelines.) (work fine with hyperv and 8g)

*** Transfer App Artifact from Docker container.

[error]Cannot find path 'c:\Run\DevOps\Abakion_Document Customizer_16.1.202034.965app.app' because it does not exist.

I only change these settings in the yaml

Removed container_restart: 'always' docker_parameters: '--isolation=hyperv'

Changed memory_gb: '16'

[waldo1001]

Can you post the complete logs as well?

[MortenRa]

2020-08-21T08:20:11.0027871Z ##[section]Starting: ALOps Compile Extension: App 2020-08-21T08:20:11.0279765Z ============================================================================== 2020-08-21T08:20:11.0280195Z Task : ALOps App Compiler 2020-08-21T08:20:11.0280498Z Description : Compiles NAV Apps 2020-08-21T08:20:11.0280792Z Version : 1.432.1571 2020-08-21T08:20:11.0281039Z Author : Hodor 2020-08-21T08:20:11.0281388Z Help : Compile a Business Central extension from AL code. 2020-08-21T08:20:11.0281910Z ============================================================================== 2020-08-21T08:20:18.1476630Z Validate configuration 2020-08-21T08:20:19.0254922Z ALOps License: 2020-08-21T08:20:19.3550249Z ALOps License: 2020-08-21T08:20:19.3550697Z 2020-08-21T08:20:19.3551267Z collectionUID : **** 2020-08-21T08:20:19.3552976Z projectUID : **** 2020-08-21T08:20:19.3553598Z licenseUID : **** 2020-08-21T08:20:19.3554354Z description : ALOps 2020-08-21T08:20:19.3554641Z 2020-08-21T08:20:19.3555981Z 2020-08-21T08:20:19.3556242Z 2020-08-21T08:20:19.3556479Z 2020-08-21T08:20:19.3819151Z Importing required PS-Functions 2020-08-21T08:20:27.6822177Z Initiate Docker Session 2020-08-21T08:20:28.1572415Z Set Docker Container ErrorActionPreference = Stop 2020-08-21T08:20:29.3583235Z Import NAV/BC Management DLL's 2020-08-21T08:20:29.3647490Z Loading assemblies for ServiceTier [BC]. 2020-08-21T08:20:29.9806656Z Loading assemblies from: [C:\program files\microsoft dynamics nav\160\service] 2020-08-21T08:20:29.9807838Z Selected module: [C:\program files\microsoft dynamics nav\160\service\Microsoft.Dynamics.Nav.Apps.Management.dll] 2020-08-21T08:20:29.9808880Z Loading module: [C:\program files\microsoft dynamics nav\160\service\Microsoft.Dynamics.Nav.Apps.Management.dll] 2020-08-21T08:20:29.9809877Z Selected module: [C:\program files\microsoft dynamics nav\160\service\Microsoft.Dynamics.Nav.Management.dll] 2020-08-21T08:20:29.9810878Z Loading module: [C:\program files\microsoft dynamics nav\160\service\Microsoft.Dynamics.Nav.Management.dll] 2020-08-21T08:20:30.3730047Z Setting up AL Compiler [ALLanguage] 2020-08-21T08:20:30.9230149Z Downloading AL Compiler [ALLanguage] 2020-08-21T08:20:30.9429686Z Get Extension from Local-Disk...[C:\Run\ALLanguage.vsix] 2020-08-21T08:20:30.9436284Z Resolving Path [C:\Run\ALLanguage.vsix] 2020-08-21T08:20:31.2345698Z VSIX Path Resolved: [C:\Run\ALLanguage.vsix] 2020-08-21T08:20:31.2378628Z Unpacking Extension to folder... 2020-08-21T08:21:10.9611759Z Loading Json Object: [c:\Run\Microsoft.al\al-ALLanguage\extension\package.json] 2020-08-21T08:21:10.9906271Z Using AL Compiler [vALLanguage] located [C:\Run\Microsoft.al\al-ALLanguage\extension\bin\alc.exe] 2020-08-21T08:21:10.9919785Z 2020-08-21T08:21:10.9935713Z Working folder: C:\Agent_work\30\s 2020-08-21T08:21:10.9943512Z 2020-08-21T08:21:10.9953966Z Copy Working folder to Docker Container: e25cb6ca1405eed3e1f2ce2623b5e849417d8c989f4002ded87540a7644f6ece 2020-08-21T08:21:10.9971090Z Docker Working Folder: c:\Run\DevOps\ 2020-08-21T08:21:11.0008272Z Copy from [C:\Agent_work\30\s] to [c:\Run\DevOps] 2020-08-21T08:21:11.0065144Z Source File setup completed 2020-08-21T08:21:12.6178811Z Target Project: \app\app.json 2020-08-21T08:21:12.6924477Z Check for app.json files: c:\Run\DevOps\ 2020-08-21T08:21:12.7825714Z Multiple projects found: 2020-08-21T08:21:12.8199275Z - C:\Run\DevOps\app 2020-08-21T08:21:12.8207886Z - C:\Run\DevOps\test 2020-08-21T08:21:12.8215430Z Retrieving App-Info [\app\app.json] 2020-08-21T08:21:12.8251534Z Loading Json Object: [c:\Run\DevOps\app\app.json] 2020-08-21T08:21:12.8275397Z App.ID = 35425804-ddcf-4013-9c67-dae40ca7890e 2020-08-21T08:21:12.8291081Z App.Name = Document Customizer 2020-08-21T08:21:12.8388597Z App.Publisher = Abakion 2020-08-21T08:21:12.8458278Z App.Version = 16.1.202033.1 2020-08-21T08:21:12.8646800Z App.Platform = 16.0.0.0 2020-08-21T08:21:12.8937394Z App.Application = 2020-08-21T08:21:12.8944910Z 2020-08-21T08:21:12.9675302Z Match: yyyyWW 2020-08-21T08:21:12.9997392Z Setting Managed App Version [c:\Run\DevOps\app\app.json] => 16.1.202034.965 2020-08-21T08:21:13.1232423Z 2020-08-21T08:21:13.3740163Z Working Direcotry: [c:\Run\DevOps] 2020-08-21T08:21:13.3800004Z Target Project: [\app\app.json] 2020-08-21T08:21:13.3806992Z Project Directory: [c:\Run\DevOps\app] 2020-08-21T08:21:13.3811465Z VSCode Settings: [c:\Run\DevOps\app.vscode\settings.json] 2020-08-21T08:21:13.3845820Z VSCode-Settings json found. 2020-08-21T08:21:13.3906857Z Loading Json Object: [c:\Run\DevOps\app.vscode\settings.json] 2020-08-21T08:21:13.4124556Z Using Ruleset: [c:\Run\DevOps\app.codeAnalysis\SCB720.ruleset.json] 2020-08-21T08:21:13.4128414Z 2020-08-21T08:21:14.4215943Z Resolve Docker Credentials 2020-08-21T08:21:14.4380144Z Setup BC Username/Password from Container info 2020-08-21T08:21:14.4452141Z Download Symbols 2020-08-21T08:21:15.3333537Z Working Dir: c:\Run\DevOps\ 2020-08-21T08:21:15.6337589Z Loading assemblies from: [C:\program files\microsoft dynamics nav\160\service] 2020-08-21T08:21:15.7732869Z PackageCache path created [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:15.7780533Z 2020-08-21T08:21:15.7994802Z Propagated Dependency: Microsoft - System Application - 1.0.0.0 2020-08-21T08:21:15.8079566Z Propagated Dependency: Microsoft - Base Application - 16.0.0.0 2020-08-21T08:21:15.8111865Z Propagated Dependency: Abakion - Abakion App Manager - 15.1.0.0 2020-08-21T08:21:15.8115755Z Download System-Symbols for [16.0.0.0] 2020-08-21T08:21:15.8928560Z Downloading NAV Symbols for [System v16.0.0.0] 2020-08-21T08:21:15.8942884Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System&versionText=16.0.0.0&tenant=default] 2020-08-21T08:21:15.9487563Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:15.9547824Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System&versionText=16.0.0.0&tenant=default 2020-08-21T08:21:16.8170350Z Content-Disposition: attachment; filename=Microsoft_System_16.0.14073.14279.app 2020-08-21T08:21:16.8697418Z Saving symbol file [Microsoft_System_16.0.14073.14279.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:16.9026071Z Read App Manifest: [c:\Run\DevOps\PackageCache\Microsoft_System_16.0.14073.14279.app]. 2020-08-21T08:21:16.9986565Z 2020-08-21T08:21:17.0022909Z Check for Depency Apps 3 2020-08-21T08:21:17.0068397Z Download Application-Symbols for [System Application v1.0.0.0] 2020-08-21T08:21:17.0158488Z Downloading NAV Symbols for [System Application v1.0.0.0] 2020-08-21T08:21:17.0162196Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=1.0.0.0&tenant=default] 2020-08-21T08:21:17.0190537Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:17.0202479Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=1.0.0.0&tenant=default 2020-08-21T08:21:17.1713624Z Content-Disposition: attachment; filename="Microsoft_System Application_16.3.14085.14363.app" 2020-08-21T08:21:17.1808198Z Saving symbol file [Microsoft_System Application_16.3.14085.14363.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:17.1951573Z Read App Manifest: [c:\Run\DevOps\PackageCache\Microsoft_System Application_16.3.14085.14363.app]. 2020-08-21T08:21:17.2064892Z 2020-08-21T08:21:17.2126710Z Download Application-Symbols for [Base Application v16.0.0.0] 2020-08-21T08:21:17.2337870Z Downloading NAV Symbols for [Base Application v16.0.0.0] 2020-08-21T08:21:17.2345479Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=Base+Application&versionText=16.0.0.0&tenant=default] 2020-08-21T08:21:17.2361667Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:17.2378567Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=Base+Application&versionText=16.0.0.0&tenant=default 2020-08-21T08:21:17.2680668Z Content-Disposition: attachment; filename="Microsoft_Base Application_16.3.14085.14363.app" 2020-08-21T08:21:17.2716944Z Saving symbol file [Microsoft_Base Application_16.3.14085.14363.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:17.3626608Z Read App Manifest: [c:\Run\DevOps\PackageCache\Microsoft_Base Application_16.3.14085.14363.app]. 2020-08-21T08:21:17.5666917Z Propagated Dependency: Microsoft - System Application - 16.3.0.0 2020-08-21T08:21:17.5948057Z 2020-08-21T08:21:17.6027281Z Download Application-Symbols for [Abakion App Manager v15.1.0.0] 2020-08-21T08:21:17.6033316Z Downloading NAV Symbols for [Abakion App Manager v15.1.0.0] 2020-08-21T08:21:17.6039628Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Abakion&appName=Abakion+App+Manager&versionText=15.1.0.0&tenant=default] 2020-08-21T08:21:17.6047296Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:17.6066983Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Abakion&appName=Abakion+App+Manager&versionText=15.1.0.0&tenant=default 2020-08-21T08:21:17.6589413Z Content-Disposition: attachment; filename="Abakion_Abakion App Manager_16.1.202034.921.app" 2020-08-21T08:21:17.6625354Z Saving symbol file [Abakion_Abakion App Manager_16.1.202034.921.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:17.6778948Z Read App Manifest: [c:\Run\DevOps\PackageCache\Abakion_Abakion App Manager_16.1.202034.921.app]. 2020-08-21T08:21:17.6938579Z Propagated Dependency: Microsoft - System Application - 16.0.0.0 2020-08-21T08:21:17.6970107Z 2020-08-21T08:21:17.7065220Z Download Application-Symbols for [System Application v16.3.0.0] 2020-08-21T08:21:17.7137077Z Downloading NAV Symbols for [System Application v16.3.0.0] 2020-08-21T08:21:17.7168875Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=16.3.0.0&tenant=default] 2020-08-21T08:21:17.7206864Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:17.7213898Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=16.3.0.0&tenant=default 2020-08-21T08:21:17.7595487Z Content-Disposition: attachment; filename="Microsoft_System Application_16.3.14085.14363.app" 2020-08-21T08:21:17.7648647Z Saving symbol file [Microsoft_System Application_16.3.14085.14363.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:17.7747758Z Read App Manifest: [c:\Run\DevOps\PackageCache\Microsoft_System Application_16.3.14085.14363.app]. 2020-08-21T08:21:17.7788457Z 2020-08-21T08:21:17.7803643Z Download Application-Symbols for [System Application v16.0.0.0] 2020-08-21T08:21:17.7844923Z Downloading NAV Symbols for [System Application v16.0.0.0] 2020-08-21T08:21:17.7854122Z Downloading NAV Symbols from [https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=16.0.0.0&tenant=default] 2020-08-21T08:21:17.7857309Z Downloading NAV Symbols with BasicAuthentication [] 2020-08-21T08:21:17.7858915Z ##[command]Invoke-RestMethod -Method Get -Uri https://e25cb6ca1405:7049/BC/dev/packages?publisher=Microsoft&appName=System+Application&versionText=16.0.0.0&tenant=default 2020-08-21T08:21:17.8347336Z Content-Disposition: attachment; filename="Microsoft_System Application_16.3.14085.14363.app" 2020-08-21T08:21:17.8399727Z Saving symbol file [Microsoft_System Application_16.3.14085.14363.app] to [c:\Run\DevOps\PackageCache] 2020-08-21T08:21:17.8508313Z Read App Manifest: [c:\Run\DevOps\PackageCache\Microsoft_System Application_16.3.14085.14363.app]. 2020-08-21T08:21:17.8605973Z 2020-08-21T08:21:18.8350906Z Starting AL Compiler... 2020-08-21T08:21:18.8370589Z App.Name = Document Customizer 2020-08-21T08:21:18.8377290Z App.Publisher = Abakion 2020-08-21T08:21:18.8403744Z App.Compiler Path= C:\Run\Microsoft.al\al-ALLanguage\extension\bin\alc.exe 2020-08-21T08:21:18.8500903Z Checking for Assemblies in App 2020-08-21T08:21:18.8995250Z Add Windows Assembly Path [C:\Windows\assembly] 2020-08-21T08:21:18.9088892Z ##[command]& C:\Run\Microsoft.al\al-ALLanguage\extension\bin\alc.exe /packagecachepath:"c:\Run\DevOps\PackageCache" /project:"c:\Run\DevOps\app" /out:"c:\Run\DevOps\Abakion_Document Customizer_16.1.202034.965app.app" /fullpaths /ruleset:"c:\Run\DevOps\app.codeAnalysis\SCB720.ruleset.json" /assemblyprobingpaths:C:\Windows\assembly 2020-08-21T08:21:20.6764551Z AL Compiler Exit Code: -1073741502 2020-08-21T08:21:20.6770796Z AL Compiler raw logs: Start 2020-08-21T08:21:20.6811862Z 2020-08-21T08:21:20.6817487Z AL Compiler raw logs: End 2020-08-21T08:21:20.6828933Z Start processing compile results. 2020-08-21T08:21:20.6896759Z Compile OK. 2020-08-21T08:21:20.6919980Z Start processing compile results done. 2020-08-21T08:21:20.6948323Z Transfer App Artifact from Docker container. 2020-08-21T08:21:21.1014219Z ##[error]Cannot find path 'c:\Run\DevOps\Abakion_Document Customizer_16.1.202034.965app.app' because it does not exist. 2020-08-21T08:21:21.4420256Z ##[section]Async Command Start: Add Build Tag 2020-08-21T08:21:21.4421226Z Build '965' has following tags now: mcr.microsoft.com/businesscentral/sandbox-latest-ltsc2019, AL.Compiler.ALLanguage, BC.Platform.16.0.14073.14279, BC.Version.16.3.14085.14363, App.Abakion App Manager.16.1.202034.921 2020-08-21T08:21:21.4421883Z ##[section]Async Command End: Add Build Tag 2020-08-21T08:21:21.4422832Z ##[section]Async Command Start: Update Build Number 2020-08-21T08:21:21.4423457Z Update build number to 16.1.202034.965 for build 965 2020-08-21T08:21:21.4423806Z ##[section]Async Command End: Update Build Number 2020-08-21T08:21:21.4425120Z ##[section]Finishing: ALOps Compile Extension: App

[AdminHodor]

Dear @MortenRa ,

The logs you posted "*** AL Compiler Exit Code: -1073741502" are related to Hyper-V isolation, which is another topic.

Could you provide the logs from pipeline where it hangs on "Starting SQL Server" ? This issue we never came across, I'm curious about whats going on there. Although it will very probably not be ALOps related. I'm guessing we'll have to look at hardware/virtualization platform and/or resources. Smells like an edge-case concerning resources.

How is the Docker machine provisioned? Is running bare-metal ? I presume it's probably also virtualised on another platform. For example, we run out Docker hosts on VMWare platform, others run it on Hyper-V, some bare-metal. When it is running virtualized you should take into account that the resources that are assigned to the machine are not guaranteed to be available. I can for example easily provision 4 docker hosts with 16GB vRAM on a host that only has 32GB RAM. To guarantee the resources are available, the VMs should be provisioned with "reserved" resourced. Not saying this is the case in your scenario, but definitely worth the investigation to check if the host-system is not over-provisioned.

Most of the cases (+80%) where we see docker containers freaking out at random for no reason it resource related (RAM/Disk).

Kind regards,

[waldo1001]

We're about to test this more specifically (hardware just arrived - so .. "soon" ;-)).

[waldo1001]

Hi @MortenRa ,

we're sure it's an infrastructure related problem. Is it possible to answer the above questions (or close the issue if not problem anymore)

[MortenRa]

Currently running fine with isolation=hyperv' enabled, so lets close this issue, it might as you say be related to infrastructure as this is a VM but I have tried to scale it with more RAM.