search

jenkinsci / azure-vm-agents-plugin

This repo is for azure vm agents plugin for jenkins. Azure devops CICD is the team which owns it for now
https://plugins.jenkins.io/azure-vm-agents/
46 stars 102 forks source link

Init script for Windows no longer working #431

Closed bsloan-icl closed 1 year ago

bsloan-icl commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.387.3 OS: Linux - 5.15.0-1035-azure Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ansicolor:1.0.2 apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 atlassian-jira-software-cloud:2.0.7 audit-trail:333.vb_e1b_b_0f1238c authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.0 azure-credentials:254.v64da_8176c83a azure-sdk:132.v62b_48eb_6f32f azure-vm-agents:859.v7213476e4fea_ bitbucket:223.vd12f2bca5430 bitbucket-oauth:0.13 bootstrap5-api:5.3.0-1 bouncycastle-api:2.28 branch-api:2.1109.vdf225489a_16d build-name-setter:2.2.0 build-timeout:1.31 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloud-stats:267.v577e3742c282 cloudbees-bitbucket-branch-source:805.v7f97d29dc0f5 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:100.v2f6722292ee8 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ configuration-as-code:1647.ve39ca_b_829b_42 credentials:1254.vb_96f366e7b_a_d credentials-binding:604.vb_64480b_c56ca_ dark-theme:315.va_22e7d692ea_a display-url-api:2.3.7 durable-task:507.v050055d0cb_dd echarts-api:5.4.0-5 email-ext:2.98 extended-read-permission:53.v6499940139e5 favorite:2.4.2 folder-properties:1.2.1 font-awesome-api:6.4.0-1 git:5.1.0 git-client:4.4.0 github:1.37.1 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1725.vd391eef681a_e groovy:453.vcdb_a_c5c99890 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:233.vdc1a_ec702cff javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jersey2-api:2.39.1-2 jira:3.10 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.84 jobConfigHistory:1212.vd4470d08ff12 jquery3-api:3.7.0-1 jsch:0.2.8-65.v052c39de79b_2 junit:1207.va_09d5100410f kubernetes:3937.vd7b_82db_e347b_ kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 kubernetes-credentials-provider:1.211.vc236a_f5a_2f3c ldap:682.v7b_544c9d1512 localization-support:1.2 mailer:457.v3f72cb_e015e5 matrix-project:789.v57a_725b_63c79 maven-plugin:3.22 mercurial:1260.vdfb_723cdcc81 metrics:4.2.18-439.v86a_20b_a_8318b_ mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ naginator:1.19.2 okhttp-api:4.11.0-145.vcb_8de402ef81 parameterized-scheduler:1.2 pipeline-build-step:496.v2449a_9a_221f2 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2141.v5402e818a_779 pipeline-model-definition:2.2141.v5402e818a_779 pipeline-model-extensions:2.2141.v5402e818a_779 pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2141.v5402e818a_779 pipeline-stage-view:2.32 plain-credentials:143.v1b_df8b_d3b_e48 plugin-usage-plugin:4.0 plugin-util-api:3.3.0 resource-disposer:0.22 role-strategy:633.v836e5b_3e80a_5 run-condition:1.5 sauce-ondemand:1.212 scm-api:672.v64378a_b_20c60 script-security:1251.vfe552ed55f8d snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 splunk-devops:1.10.1 ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.303.vefc7119b_ec23 structs:324.va_f5d6774f3a_d theme-manager:193.vcef22f6c5f2b_ token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 uipath-automation-package:3.0 variant:59.vf075fe829ccb versioncolumn:162.v85841b_0790d5 workflow-aggregator:596.v8c21c963d92d workflow-api:1213.v646def1087f9 workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3691.v28b_14c465a_b_b_ workflow-durable-task-step:1247.v7f9dfea_b_4fd0 workflow-job:1308.v58d48a_763b_31 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Jenkins 2.387.3 Docker image deployed on Kubernetes (AKS). Agent: Windows Server 2019 Datacenter.

Reproduction steps

  1. Configure the Azure VM Agent plugin as follows. The init script is basically the same as https://raw.githubusercontent.com/jenkinsci/azure-vm-agents-plugin/master/docs/init-scripts/windows-ssh.ps1 but with maven removed and python and gh-cli added.

    - templateName: "azure-win-vm-agents-2019"
templateDesc: "Azure Windows DataCenter 2019 VM agents"
maxVirtualMachinesLimit: 10
agentWorkspace: "C:\\Jenkins"
labels: "windows-vm-2019"
location: "UK South"
virtualMachineSize: "Standard_B2s"
storageAccountType: "Standard_LRS"
storageAccountNameReferenceType: "existing"
existingStorageAccountName: "devictjenkins001blobsa"
diskType: "managed"
retentionStrategy: 
azureVMCloudRetentionStrategy:
  idleTerminationMinutes: 20
usageMode: EXCLUSIVE
maximumDeploymentSize: 10
imageReference:
offer: "WindowsServer"
publisher: "MicrosoftWindowsServer"
sku: "2019-Datacenter"
version: "latest"
imageTopLevelType: "advanced"
osType: "Windows"
launcher:
ssh:
  preInstallSsh: true
initScript: |-
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
choco install -y adoptopenjdk11
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
java -version
# Install Git
choco install -y git
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
#Disable git credential manager, get more details in https://support.cloudbees.com/hc/en-us/articles/221046888-Build-Hang-or-Fail-with-Git-for-Windows
git config --system --unset credential.helper
git --version
# Install Python
choco install -y python
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
python --version
# Install GitHub CLI
choco install -y gh
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine") + ";" + [System.Environment]::GetEnvironmentVariable("Path","User")
gh version
executeInitScriptAsRoot: true
doNotUseMachineIfInitFails: true
virtualNetworkName: "dev-spoke-vn"
virtualNetworkResourceGroupName: "dev-spoke-rg"
subnetName: "dev-jenkins-sn"
usePrivateIP: true
javaPath: "java"
noOfParallelJobs: 1
credentialsId: "azure-vm-login"

  2. Try spinning up a new agent using the config above and the agent will be created but the init script will fail

Expected Results

The Azure Windows VM Agent should be provisioned and the init script should be ran successfully. This configuration was working fine until about 2 weeks ago when we noticed Windows Server 2019 VMs were no longer being launched due to the init script failing. We've tried Windows Server 2016 but same issue. Windows Server 2022 works fine.

I believe Windows Server 2019 comes with .NET 4.7 installed by default. The Chocolatey docs specify a minimum of 4.5 is required to install Chocolatey so I'm not sure why it's attempting to download a later version of .NET because 4.7 should be sufficient.

Actual Results

Looking at the logs, it appears to fail because .NET 4.8 is not found on the local machine. Chocolatey attempts to download .NET 4.8 but because a reboot is required the init script fails.

Volume in drive C is Windows
 Volume Serial Number is 184F-E00E

 Directory of C:\

File Not Found
Forcing web requests to allow TLS v1.2 (Required for requests to Chocolatey.org)
Getting latest version of the Chocolatey package for download.
Not using proxy.
Getting Chocolatey from https://community.chocolatey.org/api/v2/package/chocolatey/2.0.0.
Downloading https://community.chocolatey.org/api/v2/package/chocolatey/2.0.0 to C:\Users\azure-vm-agent\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip
Not using proxy.
Extracting C:\Users\azure-vm-agent\AppData\Local\Temp\chocolatey\chocoInstall\chocolatey.zip to C:\Users\azure-vm-agent\AppData\Local\Temp\chocolatey\chocoInstall
Installing Chocolatey on the local machine
The registry key for .Net 4.8 was not found or this is forced
Downloading 'https://download.visualstudio.microsoft.com/download/pr/2d6bb6b2-226a-4baa-bdec-798822606ff1/8494001c276a4b96804cde7829c04d7f/ndp48-x86-x64-allos-enu.exe' to 'C:\Users\azure-vm-agent\AppData\Local\Temp\ndp48-x86-x64-allos-enu.exe' - the installer is 100+ MBs, so this could take a while on a slow connection.
Installing 'C:\Users\azure-vm-agent\AppData\Local\Temp\ndp48-x86-x64-allos-enu.exe' - this may take awhile with no output.
.NET Framework 4.8 was installed, but a reboot is required. 
 Please reboot the system and try to install/upgrade Chocolatey again.
At 
C:\Users\azure-vm-agent\AppData\Local\Temp\chocolatey\chocoInstall\tools\chocolateysetup.psm1:815 
char:11
+           throw ".NET Framework 4.8 was installed, but a reboot is re ...
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (.NET Framework ...ocolatey again.:String) [], Run 
   timeException
    + FullyQualifiedErrorId : .NET Framework 4.8 was installed, but a reboot is required. 
 Please reboot the system and try to install/upgrade Chocolatey again.

Anything else?

No response

bsloan-icl commented 1 year ago

Looks like a new version of Chocolatey (2.0.0) was released on 31st May which requires .NET 4.8 as a minimum. https://docs.chocolatey.org/en-us/choco/release-notes. Because the init script downloads the latest version of Chocolatey by default, it was requiring .NET 4.8 to be installed which isn't the default on Windows Server 2019.

Adding $env:chocolateyVersion = '1.4.0' to the beginning on the init script resolved this issue.