search

jenkinsci / azure-vm-agents-plugin

This repo is for azure vm agents plugin for jenkins. Azure devops CICD is the team which owns it for now
https://plugins.jenkins.io/azure-vm-agents/
45 stars 101 forks source link

VM not deleted if failure occurs in #terminateVirtualMachine #267

Open henti opened 3 years ago

henti commented 3 years ago

Version report

Jenkins and plugins versions report:

Jenkins : 
Debian 8 64bit
Openjdk version "1.8.0_171"
Jenkins 2.263.4 
Metrics Plugin (metrics): 4.0.2.7
PAM Authentication plugin (pam-auth): 1.6
Pipeline: Model API (pipeline-model-api): 1.7.2
Run Condition Plugin (run-condition): 1.5
Pipeline Utility Steps (pipeline-utility-steps): 2.6.1
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0
ECharts API Plugin (echarts-api): 4.9.0-2
Popper.js API Plugin (popper-api): 1.16.0-7
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 2.1
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 1.4
JQuery3 API Plugin (jquery3-api): 3.5.1-2
jQuery plugin (jquery): 1.12.4-1
Authentication Tokens API Plugin (authentication-tokens): 1.4
Credentials Binding Plugin (credentials-binding): 1.24
Icon Shim Plugin (icon-shim): 2.0.3
Pipeline: Input Step (pipeline-input-step): 2.12
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
GitHub Branch Source Plugin (github-branch-source): 2.9.3
Command Agent Launcher Plugin (command-launcher): 1.5
Plugin Utilities API Plugin (plugin-util-api): 1.6.1
build log file size checker plugin (logfilesizechecker): 1.5
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
Matrix Project Plugin (matrix-project): 1.18
Resource Disposer Plugin (resource-disposer): 0.14
Build Timeout (build-timeout): 1.20
Pipeline: Declarative (pipeline-model-definition): 1.7.2
Jackson 2 API Plugin (jackson2-api): 2.12.0
Trilead API Plugin (trilead-api): 1.0.13
Copy Artifact Plugin (copyartifact): 1.45.3
SCM API Plugin (scm-api): 2.6.4
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.7.2
Priority Sorter Plugin (PrioritySorter): 3.6.0
Pipeline: Stage Step (pipeline-stage-step): 2.5
Azure Commons Plugin (azure-commons): 1.0.5
Email Extension Plugin (email-ext): 2.80
Pipeline: Multibranch (workflow-multibranch): 2.22
Checks API plugin (checks-api): 1.2.0
Gradle Plugin (gradle): 1.36
OkHttp Plugin (okhttp-api): 3.14.9
Pipeline: Groovy (workflow-cps): 2.87
Pipeline (workflow-aggregator): 2.6
Pipeline: API (workflow-api): 2.40
Script Security Plugin (script-security): 1.75
Folders Plugin (cloudbees-folder): 6.15
Build Failure Analyzer (build-failure-analyzer): 1.27.1
MSBuild Plugin (msbuild): 1.29
GIT server Plugin (git-server): 1.9
SSH Agent Plugin (ssh-agent): 1.20
Branch API Plugin (branch-api): 2.6.2
promoted builds plugin (promoted-builds): 3.7
Matrix Authorization Strategy Plugin (matrix-auth): 2.6.4
Pipeline: Stage View Plugin (pipeline-stage-view): 2.19
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.17
SSH Credentials Plugin (ssh-credentials): 1.18.1
SSH Build Agents plugin (ssh-slaves): 1.31.5
Timestamper (timestamper): 1.11.8
Job DSL (job-dsl): 1.77
Ant Plugin (ant): 1.11
Pipeline: SCM Step (workflow-scm-step): 2.11
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.10
Bootstrap 4 API Plugin (bootstrap4-api): 4.5.3-1
External Monitor Job Type Plugin (external-monitor-job): 1.7
Mailer Plugin (mailer): 1.32.1
GitHub API Plugin (github-api): 1.117
Azure Credentials (azure-credentials): 4.0.5
WMI Windows Agents Plugin (windows-slaves): 1.7
Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.1
Font Awesome API Plugin (font-awesome-api): 5.15.1-1
Conditional BuildStep (conditional-buildstep): 1.4.1
JSch dependency plugin (jsch): 0.1.55.2
Node and Label parameter plugin (nodelabelparameter): 1.7.2
Maven Integration plugin (maven-plugin): 3.8
Pipeline: Basic Steps (workflow-basic-steps): 2.22
Token Macro Plugin (token-macro): 2.13
Snakeyaml API Plugin (snakeyaml-api): 1.27.0
Rebuilder (rebuild): 1.31
Workspace Cleanup Plugin (ws-cleanup): 0.38
jQuery UI plugin (jquery-ui): 1.0.2
Pipeline: REST API Plugin (pipeline-rest-api): 2.19
Async Http Client (async-http-client): 1.9.40.0
Pipeline: Step API (workflow-step-api): 2.23
Credentials Plugin (credentials): 2.3.14
Structs Plugin (structs): 1.20
LDAP Plugin (ldap): 1.26
Git client plugin (git-client): 3.6.0
Docker Pipeline (docker-workflow): 1.25
Plain Credentials Plugin (plain-credentials): 1.7
MapDB API Plugin (mapdb-api): 1.0.9.0
Javadoc Plugin (javadoc): 1.6
Variant Plugin (variant): 1.4
GitHub Organization Folder Plugin (github-organization-folder): 1.6
GitHub Pull Request Builder (ghprb): 1.42.1
GitHub Authentication plugin (github-oauth): 0.33
Azure VM Agents (azure-vm-agents): 1.5.1
GitHub plugin (github): 1.32.0
Display URL API (display-url-api): 2.3.4
Subversion Plug-in (subversion): 2.13.2
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Parameterized Trigger plugin (parameterized-trigger): 2.39
Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
Pipeline: Nodes and Processes (workflow-durable-task-step): 2.37
Pipeline: Build Step (pipeline-build-step): 2.13
Pipeline: Supporting APIs (workflow-support): 3.7
Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
Groovy (groovy): 2.3
Job Configuration History Plugin (jobConfigHistory): 2.26
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Durable Task Plugin (durable-task): 1.35
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.7.2
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.13-1.0
JUnit Plugin (junit): 1.48
Docker Commons Plugin (docker-commons): 1.17
Build Alias Setter Plugin (build-alias-setter): 0.4
Git plugin (git): 4.5.2
Pipeline: Job (workflow-job): 2.40
bouncycastle API Plugin (bouncycastle-api): 2.18
Cloud Statistics Plugin (cloud-stats): 0.25
Lockable Resources plugin (lockable-resources): 2.10

Agent :
Debian Buster 64bit
openjdk version "1.8.0_265"

Reproduction steps

Results

Expected result:

Actual result:

Logs from Jenkins. 

2021-04-01 21:54:37.158+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: Deprovision called for agent jenkins-8core-workerf73650, for reason: Node is being deleted by Jenkins after idle timeout
2021-04-01 21:54:37.630+0000 [id=811938]        INFO    o.j.p.workflow.job.WorkflowRun#finish: ODASA #19127 completed: SUCCESS
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: Template terminate script,
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.azure.vmagent.AzureVMAgent#deprovision: AzureVMAgent: deprovision: skipping terminate script execution.
2021-04-01 21:54:57.174+0000 [id=812700]        INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for jenkins-8core-workerf73650
2021-04-01 21:54:57.175+0000 [id=812700]        WARNING c.m.a.v.AzureVMManagementServiceDelegate#terminateVirtualMachine: AzureVMManagementServiceDelegate: terminateVirtualMachine: while deleting VM
java.net.UnknownHostException: management.azure.com
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at okhttp3.Dns$1.lookup(Dns.java:40)
        at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:185)
        at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:149)
        at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84)
        at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:214)
        at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
        at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:54)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.azure.vmagent.AzureVMAgentPlugin$AzureTelemetryInterceptor.intercept(AzureVMAgentPlugin.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:254)
        at okhttp3.RealCall.execute(RealCall.java:92)
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:186)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
        at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.subscribe(Observable.java:10423)
        at rx.Observable.subscribe(Observable.java:10390)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
Caused: java.lang.RuntimeException
        at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
        at rx.observables.BlockingObservable.last(BlockingObservable.java:226)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:110)
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.GroupableResourcesImpl.getByResourceGroup(GroupableResourcesImpl.java:34)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.virtualMachineExists(AzureVMManagementServiceDelegate.java:930)
Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException
        at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54)
        at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.virtualMachineExists(AzureVMManagementServiceDelegate.java:932)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.terminateVirtualMachine(AzureVMManagementServiceDelegate.java:1728)
        at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.terminateVirtualMachine(AzureVMManagementServiceDelegate.java:1698)
        at com.microsoft.azure.vmagent.AzureVMAgent.deprovision(AzureVMAgent.java:654)
        at com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1.call(AzureVMCloudRetensionStrategy.java:102)
        at com.microsoft.azure.vmagent.AzureVMCloudRetensionStrategy$1.call(AzureVMCloudRetensionStrategy.java:91)
        at com.microsoft.azure.vmagent.retry.RetryTask.call(RetryTask.java:49)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
timja commented 3 years ago

What makes this an issue with this plugin?

It appears to be an issue with your DNS, does the plugin later retry deleting the VM?

henti commented 3 years ago

That is the bug,

The VM is kept in "off-line" mode in Jenkins but never deleted. The VM is not shutdown and the plugin does not attempt to delete the VM at a later stage.