search

jenkinsci / azure-vm-agents-plugin

This repo is for azure vm agents plugin for jenkins. Azure devops CICD is the team which owns it for now
https://plugins.jenkins.io/azure-vm-agents/
45 stars 101 forks source link

Issue spinning of Jenkins Agent from updated Azure image #372

Closed limeman40 closed 2 years ago

limeman40 commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.359 OS: Linux - 5.4.0-1069-azure --- Office-365-Connector:4.17.0 ace-editor:1.1 ansible:1.1 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 async-http-client:1.9.40.0 authentication-tokens:1.4 azure-acs:1.0.4 azure-ad:241.vb_e5cd7c35b_2e azure-app-service:1.0.2 azure-artifact-manager:97.v074e1332e88d azure-cli:0.9 azure-commons:1.1.3 azure-container-agents:241.va_780fa_dc374a_ azure-container-registry-tasks:0.6.5 azure-credentials:216.ve0b_4a_485ffc2 azure-credentials-ext:1.0 azure-function:0.3.3 azure-keyvault:136.veb1d1296db97 azure-sdk:118.v43f74dd9ca_dc azure-vm-agents:822.v3a18fc3d2de1 azure-vmss:0.2.4 badge:1.9.1 blackduck-detect:7.0.0 block-queued-job:0.2.0 blueocean-bitbucket-pipeline:1.25.5 blueocean-commons:1.25.5 blueocean-core-js:1.25.5 blueocean-jwt:1.25.5 blueocean-pipeline-api-impl:1.25.5 blueocean-pipeline-scm-api:1.25.5 blueocean-rest:1.25.5 blueocean-rest-impl:1.25.5 blueocean-web:1.25.5 bootstrap4-api:4.6.0-5 bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe changes-since-last-success:0.6 checks-api:1.7.4 cloud-stats:0.27 cloudbees-bitbucket-branch-source:773.v4b_9b_005b_562b_ cloudbees-folder:6.729.v2b_9d1a_74d673 command-launcher:84.v4a_97f2027398 conditional-buildstep:1.4.2 config-file-provider:3.10.0 copyartifact:1.46.4 credentials:1139.veb_9579fca_33b_ credentials-binding:523.vd859a_4b_122e6 crx-content-package-deployer:1.9 datadog:4.0.0 digitalocean-plugin:1.3.0 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.2.13-37.vf3411c9828b9 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 envinject:2.866.v5c0403e3d4df envinject-api:1.199.v3ce31253ed13 extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:191.v363d0d1efdf8 favorite:2.4.1 font-awesome-api:6.1.1-1 generic-webhook-trigger:1.84 git:4.11.3 git-client:3.11.0 git-parameter:0.9.17 git-server:1.11 github:1.34.4 github-api:1.303-400.v35c2d8258028 github-branch-source:1656.v77eddb_b_e95df github-pullrequest:0.4.0 gitlab-api:5.0.1-74.v44f46b_54c775 gitlab-branch-source:630.v04ca_c57fa_880 google-metadata-plugin:0.3.1 google-oauth-plugin:1.0.6 gradle:1.39.4 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.30 instance-identity:3.1 jackson2-api:2.13.3-285.vc03c0256d517 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-3 javax-mail-api:1.6.2-6 jaxb:2.3.6-1 jdk-tool:1.5 jenkins-design-language:1.25.5 jersey2-api:2.36-2 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.15-1 jobConfigHistory:1156.v536a_97b_8d649 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-4 jsch:0.1.55.2 junit:1119.1121.vc43d0fc45561 kubernetes-cd:2.3.1 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 label-linked-jobs:6.0.1 ldap:2.10 lockable-resources:2.15 mailer:414.vcc4c33714601 mapdb-api:1.0.9.0 matrix-auth:3.1.5 matrix-project:772.v494f19991984 maven-plugin:3.19 mercurial:2.16.2 metrics:4.1.6.2 mina-sshd-api-common:2.8.0-21.v493b_6b_db_22c6 mina-sshd-api-core:2.8.0-21.v493b_6b_db_22c6 momentjs:1.1.1 msbuild:1.30 nexus-jenkins-plugin:3.14.424.v8290b_b_ec62cb_ node-iterator-api:1.5.1 node-sharing-executor:2.0.8 oauth-credentials:0.5 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.8 pipeline-build-step:2.18 pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:593.va_a_fc25d520e9 pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2097.v33db_b_de764b_e pipeline-model-definition:2.2097.v33db_b_de764b_e pipeline-model-extensions:2.2097.v33db_b_de764b_e pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2097.v33db_b_de764b_e pipeline-stage-view:2.24 pipeline-utility-steps:2.13.0 plain-credentials:1.8 plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.5-2 powershell:1.7 pubsub-light:1.16 resource-disposer:0.19 run-condition:1.5 saml:2.298.vc7a_2b_3958628 scm-api:608.vfa_f971c5a_a_e9 script-security:1175.v4b_d517d6db_f0 service-fabric:1.6 shelve-project-plugin:3.2 snakeyaml-api:1.30.2-76.vc104f7ce9870 ssh:2.6.1 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:291.v8211e4f8efb_c ssh-slaves:1.821.vd834f8a_c390e ssh2easy:1.4 sshd:3.242.va_db_9da_b_26a_c3 stashNotifier:1.28 strict-crumb-issuer:2.1.0 structs:318.va_f3ccb_729b_71 synopsys-coverity:3.0.1 thinBackup:1.10 timestamper:1.18 token-macro:293.v283932a_0a_b_49 trilead-api:1.66.v49c6758b_b_360 uno-choice:2.6.3 variant:1.4 windows-azure-storage:378.vb2cb9797e1e5 windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1188.v0016b_4f29881 workflow-basic-steps:969.vc4ec3e4854b_f workflow-cps:2746.v0da_83a_332669 workflow-cps-global-lib:588.v576c103a_ff86 workflow-durable-task-step:1174.v73a_9a_17edce0 workflow-job:1206.vc48d96b_930b_2 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:625.vd896b_f445a_f8 workflow-support:833.va_1c71061486b_ ws-cleanup:0.42 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux (ubuntu 18.04) Jenkins

Windows 2019 Image

Reproduction steps

  1. Create a VM image
  2. Create a template that uses this image.
  3. Create a Jenkins job that uses the label of this image

Expected Results

That the VM gets spun up from this image and attached to Jenkins and jobs can run without issue. I having trouble using updated images from an existing image the VM seems to spin up ok but than the job fails to run with messages saying the agent is not ready.

Actual Results

The agent seems to be connected without issue yet I can not run any jobs on that agent

Anything else?

This is the error I see in the agent log

This is a Windows agent
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
java.lang.NoSuchFieldException: modifiers
    at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
    at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:35)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to winagent86d290
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at hudson.FilePath.act(FilePath.java:1277)
        at org.jenkinsci.plugins.envinject.EnvInjectComputerListener.onOnline(EnvInjectComputerListener.java:144)
        at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)
        at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:438)
        at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.launch(AzureVMAgentSSHLauncher.java:245)
        at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
        at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused: org.jenkinsci.lib.envinject.EnvInjectException
    at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:42)
    at org.jenkinsci.plugins.envinject.service.EnvInjectMasterEnvVarsSetter.call(EnvInjectMasterEnvVarsSetter.java:16)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:376)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused: java.io.IOException
    at org.jenkinsci.plugins.envinject.EnvInjectComputerListener.onOnline(EnvInjectComputerListener.java:146)
    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)
    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:438)
    at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.launch(AzureVMAgentSSHLauncher.java:245)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
timja commented 2 years ago

Appears to be this issue from a quick google: https://issues.jenkins.io/browse/JENKINS-66896

limeman40 commented 2 years ago

Hmm well there appears to be no solution for it. I was not getting this message before from your plugin which is odd to me.

I am also seeing this in the error output:

dir: cannot access 'C:\.azure-agent-init': No such file or directory fatal: not in a git directory 1 file(s) copied. java full version "17.0.1+12-LTS-39" <===[JENKINS REMOTING CAPACITY]===>Remoting version: 3028.va_a_436db_35078

timja commented 2 years ago

I assume you're running Java 17 on the agent? can you check the java version?

do you need the envinject plugin?

limeman40 commented 2 years ago

developer@jenkins:~$ java -version openjdk version "11.0.15" 2022-04-19 OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1) OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1, mixed mode, sharing)

Does your plugin not use envinject plugin?

timja commented 2 years ago

developer@jenkins:~$ java -version openjdk version "11.0.15" 2022-04-19 OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1) OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1, mixed mode, sharing)

Does your plugin not use envinject plugin?

It does not: https://plugins.jenkins.io/azure-vm-agents/#dependencies

developer@jenkins:~$ java -version

This is showing the output from a linux host, the error message indicates a windows host.

limeman40 commented 2 years ago

The windows 2019 is this.. The thing is it still connects event though it stack traces..

C:\Users\s-devops>java -version
java version "17.0.1" 2021-10-19 LTS
Java(TM) SE Runtime Environment (build 17.0.1+12-LTS-39)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.1+12-LTS-39, mixed mode, sharing)

The issue is I am trying to update this image with additional tools and that is refusing to work even after it is connected. Maybe this is a red herring and not the issue not sure at this point

timja commented 2 years ago

Can you use Java 11 instead of Java 17?

limeman40 commented 2 years ago

I can try to make an image with Java 11 sure. Let me see if that works any better

limeman40 commented 2 years ago

I made an image with Java 11 seems to work fine. You can close this as it is not a plugin issue