search

jenkinsci / azure-vm-agents-plugin

This repo is for azure vm agents plugin for jenkins. Azure devops CICD is the team which owns it for now
https://plugins.jenkins.io/azure-vm-agents/
46 stars 102 forks source link

Associated resources are not deleted from Azure after Retention Time #375

Closed shanep2300 closed 1 year ago

shanep2300 commented 2 years ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.332.1 OS: Linux - 3.10.0-1160.66.1.el7.x86_64 --- Office-365-Connector:4.16.1 ace-editor:1.1 active-directory:2.25.1 allure-jenkins-plugin:2.30.2 amazon-ecs:1.41 analysis-model-api:10.9.4 ansible:1.1 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 artifactory:3.13.2 atlassian-bitbucket-server-integration:3.2.0 audit-trail:3.10 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df aws-lambda:0.5.10 azure-ad:1.2.1 azure-cli:0.9 azure-commons:1.1.3 azure-container-agents:234.v495633ea_6725 azure-credentials:216.ve0b_4a_485ffc2 azure-sdk:106.v552de1e64d56 azure-vm-agents:810.v0a97a847315a bitbucket:223.vd12f2bca5430 bitbucket-push-and-pull-request:2.8.1 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jira:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.1044.v2c007e51b_87f browserstack-integration:1.2.3 build-environment:1.7 build-failure-analyzer:2.0.0 build-timestamp:1.0.3 build-user-vars-plugin:1.8 build-with-parameters:1.6 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 chromedriver:1.2 cloud-stats:0.27 cloudbees-bitbucket-branch-source:762.v969cfe087fc0 cloudbees-folder:6.714.v79e858ef76a_2 cobertura:1.17 code-coverage-api:2.0.4 codedeploy:1.23 command-launcher:1.6 conditional-buildstep:1.4.2 config-file-provider:3.9.0 configurationslicing:430.v966357576543 copyartifact:1.46.3 countjobs-viewstabbar:1.0.1 credentials:1087.v16065d268466 credentials-binding:1.27.1 crx-content-package-deployer:1.9 cucumber-reports:5.7.0 custom-tools-plugin:0.8 data-tables-api:1.11.4-4 datadog:4.0.0 dependency-check-jenkins-plugin:5.1.2 dependency-track:4.1.1 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.1.5.2 docker-plugin:1.2.6 docker-workflow:1.28 durable-task:495.v29cd95ec10f2 ec2:1.68 echarts-api:5.3.0-2 email-ext:2.87 envinject:2.847.vb_fa_18490fa_d6 envinject-api:1.192.va_13643e71792 extended-choice-parameter:346.vd87693c5a_86c extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:191.v363d0d1efdf8 favorite:2.4.1 file-operations:1.11 font-awesome-api:6.0.0-1 forensics-api:1.12.0 git:4.11.0 git-changelog:3.23 git-client:3.11.0 git-parameter:0.9.15 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:1598.v91207e9f9b_4a_ gradle:1.38 greenballs:1.15.1 groovy:2.4 h2-api:1.4.199 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 hashicorp-vault-plugin:336.v182c0fbaaeb7 hidden-parameter:0.0.4 htmlpublisher:1.29 ivy:2.1 jackson2-api:2.13.2-260.v43d711474c77 jacoco:3.3.1 javadoc:217.v905b_86277a_2a_ javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.3 jira:3.7 jira-trigger:1.0.1 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-3 job-dsl:1.77 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.58 ldap:2.8 lockable-resources:2.14 log-parser:2.2 mailer:408.vd726a_1130320 mapdb-api:1.0.9.0 mask-passwords:3.0 matrix-auth:2.6.6 matrix-project:758.v7a_ea_491852f3 maven-plugin:3.18 mercurial:2.16 momentjs:1.1.1 monitoring:1.91.0 msbuild:1.30 mstest:1.0.0 node-iterator-api:1.5.1 nodejs:1.5.1 nuget:1.1 nunit:0.27 okhttp-api:4.9.3-105.vb96869f8ac3a packer:1.5 pam-auth:1.7 parameter-separator:1.3 parameterized-scheduler:1.0 parameterized-trigger:2.44 periodicbackup:1.8 pipeline-build-step:2.16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-maven:3.10.0 pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:2.2075.vce74e77b_ce40 pipeline-model-definition:2.2075.vce74e77b_ce40 pipeline-model-extensions:2.2075.vce74e77b_ce40 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:2.2075.vce74e77b_ce40 pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-usage-plugin:2.2 plugin-util-api:2.16.0 popper-api:1.16.1-2 popper2-api:2.11.4-1 postgresql-api:42.3.3 powershell:1.7 prism-api:1.26.0-2 prisma-cloud-iac-scan:1.3.5 prisma-cloud-jenkins-plugin:22.01.880 pubsub-light:1.16 rebuild:1.33 remote-file:1.21 run-condition:1.5 saml:2.296.v0016349946db_ scalable-amazon-ecs:1.0 schedule-build:244.vce2a_2e926127 scm-api:595.vd5a_df5eb_0e39 script-security:1145.vb_cf6cf6ed960 simple-theme-plugin:103.va_161d09c38c7 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.29.1 sonar:2.14 splunk-devops:1.9.9 splunk-devops-extend:1.9.9 sse-gateway:1.25 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.806.v2253cedd3295 sshd:3.1.0 stashNotifier:1.26 strict-crumb-issuer:2.1.0 structs:308.v852b473a2b8c subversion:2.15.3 terraform:1.0.10 thycotic-credentials:1.0 timestamper:1.17 token-macro:285.vff7645a_56ff0 trilead-api:1.0.13 uno-choice:2.6.1 variant:1.4 warnings-ng:9.11.1 windows-azure-storage:373.v582b31a65906 windows-slaves:1.8 workflow-aggregator:2.7 workflow-api:1143.v2d42f1e9dea_5 workflow-basic-steps:941.vdfe1b_a_132c64 workflow-cps:2683.vd0a_8f6a_1c263 workflow-cps-global-lib:564.ve62a_4eb_b_e039 workflow-durable-task-step:1128.v8c259d125340 workflow-job:1174.vdcb_d054cf74a_ workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:815.vd60466279fc8 xray-for-jira-connector:1.2.1 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux Centos7

Reproduction steps

  1. Wait for retention timeout

Expected Results

  1. Node is removed from Jenkins
  2. All associated resources in Azure are removed as well

Actual Results

None of the Azure resources are removed. Have to be done manually.

For example: these nodes do not exist anymore in Jenkins

image

Anything else?

No response

cyclelabs-ryanberger commented 2 years ago

Does the service principal/managed identity that you're using to provision the VM's have Contributor access to the resource group? Maybe the delete action is failing due to permissions?

On Wed, Oct 5, 2022 at 10:13 AM shanep2300 @.***> wrote:

Jenkins and plugins versions report Environment

Jenkins: 2.332.1 OS: Linux - 3.10.0-1160.66.1.el7.x86_64

Office-365-Connector:4.16.1 ace-editor:1.1 active-directory:2.25.1 allure-jenkins-plugin:2.30.2 amazon-ecs:1.41 analysis-model-api:10.9.4 ansible:1.1 ansicolor:1.0.1 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 artifactory:3.13.2 atlassian-bitbucket-server-integration:3.2.0 audit-trail:3.10 authentication-tokens:1.4 authorize-project:1.4.0 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df aws-lambda:0.5.10 azure-ad:1.2.1 azure-cli:0.9 azure-commons:1.1.3 azure-container-agents:234.v495633ea_6725 azure-credentials:216.ve0b_4a_485ffc2 azure-sdk:106.v552de1e64d56 azure-vm-agents:810.v0a97a847315a bitbucket:223.vd12f2bca5430 bitbucket-push-and-pull-request:2.8.1 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jira:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.1044.v2c007e51b_87f browserstack-integration:1.2.3 build-environment:1.7 build-failure-analyzer:2.0.0 build-timestamp:1.0.3 build-user-vars-plugin:1.8 build-with-parameters:1.6 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 chromedriver:1.2 cloud-stats:0.27 cloudbees-bitbucket-branch-source:762.v969cfe087fc0 cloudbees-folder:6.714.v79e858ef76a_2 cobertura:1.17 code-coverage-api:2.0.4 codedeploy:1.23 command-launcher:1.6 conditional-buildstep:1.4.2 config-file-provider:3.9.0 configurationslicing:430.v966357576543 copyartifact:1.46.3 countjobs-viewstabbar:1.0.1 credentials:1087.v16065d268466 credentials-binding:1.27.1 crx-content-package-deployer:1.9 cucumber-reports:5.7.0 custom-tools-plugin:0.8 data-tables-api:1.11.4-4 datadog:4.0.0 dependency-check-jenkins-plugin:5.1.2 dependency-track:4.1.1 display-url-api:2.3.6 docker-commons:1.19 docker-java-api:3.1.5.2 docker-plugin:1.2.6 docker-workflow:1.28 durable-task:495.v29cd95ec10f2 ec2:1.68 echarts-api:5.3.0-2 email-ext:2.87 envinject:2.847.vb_fa_18490fa_d6 envinject-api:1.192.va_13643e71792 extended-choice-parameter:346.vd87693c5a_86c extended-read-permission:3.2 extensible-choice-parameter:1.8.0 external-monitor-job:191.v363d0d1efdf8 favorite:2.4.1 file-operations:1.11 font-awesome-api:6.0.0-1 forensics-api:1.12.0 git:4.11.0 git-changelog:3.23 git-client:3.11.0 git-parameter:0.9.15 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:1598.v91207e9f9b4a gradle:1.38 greenballs:1.15.1 groovy:2.4 h2-api:1.4.199 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 hashicorp-vault-plugin:336.v182c0fbaaeb7 hidden-parameter:0.0.4 htmlpublisher:1.29 ivy:2.1 jackson2-api:2.13.2-260.v43d711474c77 jacoco:3.3.1 javadoc:217.v905b_86277a2a javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.3 jira:3.7 jira-trigger:1.0.1 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-3 job-dsl:1.77 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.58 ldap:2.8 lockable-resources:2.14 log-parser:2.2 mailer:408.vd726a_1130320 mapdb-api:1.0.9.0 mask-passwords:3.0 matrix-auth:2.6.6 matrix-project:758.v7a_ea_491852f3 maven-plugin:3.18 mercurial:2.16 momentjs:1.1.1 monitoring:1.91.0 msbuild:1.30 mstest:1.0.0 node-iterator-api:1.5.1 nodejs:1.5.1 nuget:1.1 nunit:0.27 okhttp-api:4.9.3-105.vb96869f8ac3a packer:1.5 pam-auth:1.7 parameter-separator:1.3 parameterized-scheduler:1.0 parameterized-trigger:2.44 periodicbackup:1.8 pipeline-build-step:2.16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-maven:3.10.0 pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:2.2075.vce74e77b_ce40 pipeline-model-definition:2.2075.vce74e77b_ce40 pipeline-model-extensions:2.2075.vce74e77b_ce40 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:2.2075.vce74e77bce40 pipeline-stage-view:2.23 pipeline-utility-steps:2.12.0 plain-credentials:1.8 plugin-usage-plugin:2.2 plugin-util-api:2.16.0 popper-api:1.16.1-2 popper2-api:2.11.4-1 postgresql-api:42.3.3 powershell:1.7 prism-api:1.26.0-2 prisma-cloud-iac-scan:1.3.5 prisma-cloud-jenkins-plugin:22.01.880 pubsub-light:1.16 rebuild:1.33 remote-file:1.21 run-condition:1.5 saml:2.296.v0016349946db scalable-amazon-ecs:1.0 schedule-build:244.vce2a_2e926127 scm-api:595.vd5a_df5eb_0e39 script-security:1145.vb_cf6cf6ed960 simple-theme-plugin:103.va_161d09c38c7 slack:608.v19e3b_44b_b_9ff snakeyaml-api:1.29.1 sonar:2.14 splunk-devops:1.9.9 splunk-devops-extend:1.9.9 sse-gateway:1.25 ssh-agent:1.24.1 ssh-credentials:1.19 ssh-slaves:1.806.v2253cedd3295 sshd:3.1.0 stashNotifier:1.26 strict-crumb-issuer:2.1.0 structs:308.v852b473a2b8c subversion:2.15.3 terraform:1.0.10 thycotic-credentials:1.0 timestamper:1.17 token-macro:285.vff7645a_56ff0 trilead-api:1.0.13 uno-choice:2.6.1 variant:1.4 warnings-ng:9.11.1 windows-azure-storage:373.v582b31a65906 windows-slaves:1.8 workflow-aggregator:2.7 workflow-api:1143.v2d42f1e9dea_5 workflow-basic-steps:941.vdfe1b_a_132c64 workflow-cps:2683.vd0a_8f6a_1c263 workflow-cps-global-lib:564.ve62a_4eb_b_e039 workflow-durable-task-step:1128.v8c259d125340 workflow-job:1174.vdcbd054cf74a workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15bc95a workflow-support:815.vd60466279fc8 xray-for-jira-connector:1.2.1

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux Centos7 Reproduction steps

  1. Wait for retention timeout

Expected Results

  1. Node is removed from Jenkins
  2. All associated resources in Azure are removed as well

Actual Results

None of the Azure resources are removed. Have to be done manually.

For example: these nodes do not exist anymore in Jenkins [image: image] https://user-images.githubusercontent.com/16820289/194082126-54f7541d-9f99-4622-863a-d79299fd3ad8.png Anything else?

No response

— Reply to this email directly, view it on GitHub https://github.com/jenkinsci/azure-vm-agents-plugin/issues/375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOLST6XV5A26Y23XI4NNYA3WBWEIXANCNFSM6AAAAAAQ5TZXLA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

timja commented 2 years ago

Check the logs in Jenkins. It should be regularly cleaning everything in the Jenkins managed resource group that is managed by the tag it uses (roughly JenkinsManaged)

shanep2300 commented 2 years ago

@ryanberger-az I will definitely look into that!

@timja As for the logs, I'm mostly just seeing:

AzureVMCloud: getAzureAgentTemplate: Found agent template azure-linux-tf12-prod
Oct 05, 2022 3:25:19 PM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Found agent template azure-linux-tf13-prod
Oct 05, 2022 3:25:19 PM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Found agent template azure-linux-tf1x-prod
Oct 05, 2022 3:25:19 PM FINE com.microsoft.azure.vmagent.AzureVMCloud

But this popped up as I was looking into this

Started Azure VM Agents Clean Task
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Start
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Running clean with 15 minute timeout
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Beginning
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Completed
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Cleaning deployments
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Checking deployment azure-linux-tf1x-prod1005143613456
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Deployment created on 2022-10-05T14:37:08.747752700Z
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Deployment newer than timeout, keeping
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Checking deployment azure-linux-tf13-prod1005145153456
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Deployment created on 2022-10-05T14:52:45.762808300Z
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Deployment newer than timeout, keeping
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Done cleaning deployments
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Beginning
Oct 05, 2022 3:37:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
cleanLeakedResources: beginning to look at leaked resources in rg: gccoemgmt-jenkins-prod-rg-01
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
cleanLeakedResources: %d resources marked for deletion0
Oct 05, 2022 3:22:26 PM WARNING com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanLeakedResources
Failed to clean leaked resources 
Also:   java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
        at reactor.core.publisher.Flux.blockLast(Flux.java:2645)
        at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94)
java.lang.IllegalArgumentException: Parameter 'resourceGroupName' is required and cannot be null.
    at com.azure.resourcemanager.resources.implementation.GenericResourcesImpl.lambda$listByResourceGroupAsync$7(GenericResourcesImpl.java:345)
    at com.azure.core.http.rest.PagedFlux.lambda$new$2(PagedFlux.java:145)
    at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:85)
    at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.<init>(ContinuablePagedByItemIterable.java:58)
    at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:41)
    at com.azure.core.util.paging.ContinuablePagedIterable.iterator(ContinuablePagedIterable.java:106)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask.cleanLeakedResources(AzureVMAgentCleanUpTask.java:345)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask.cleanLeakedResources(AzureVMAgentCleanUpTask.java:312)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask.clean(AzureVMAgentCleanUpTask.java:600)
    at com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask.lambda$execute$1(AzureVMAgentCleanUpTask.java:610)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
cleanLeakedResources: beginning to look at leaked resources in rg: gccoemgmt-jenkins-prod-rg-02
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
cleanLeakedResources: %d resources marked for deletion0
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Completed
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Beginning
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Completed
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
End
Oct 05, 2022 3:22:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Finished Azure VM Agents Clean Task. 451 ms
Oct 05, 2022 3:23:23 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask
Started Azure VM Maintainer Pool Size
Oct 05, 2022 3:23:23 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask
Finished Azure VM Maintainer Pool Size. 0 ms
Oct 05, 2022 3:23:39 PM FINE com.microsoft.azure.vmagent.AzureVMCloud

Is that what you're referring too?

timja commented 2 years ago

Yes that will be it, it can't find the resourceGroupName for some reason which is breaking the cleanup.

Maybe check the xml files on the disk (I think config.xml), see if there's any old config in there / missing the resource group name.

shanep2300 commented 2 years ago

@timja Figured out the RG error. There was a cloud config using the wrong SP.

@ryanberger-az The SP has Contributor role at the subscription level.

cleanLeakedResources: beginning to look at leaked resources in rg: gccoemgmt-jenkins-prod-rg-01
Oct 06, 2022 12:07:26 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
cleanLeakedResources: %d resources marked for deletion0

At this current time, there are only 2 active nodes in Jenkins, but around 67 resources in that RG.

shanep2300 commented 2 years ago

Tried deleting the resources manually, not sure if it's an Azure Portal visual bug or something else, but some resources would come back and would have to be deleted a second time.

Either way, manually deleting older resources didn't change anything.

Is there a bug in the output string/syntax? Output: cleanLeakedResources: %d resources marked for deletion0

Noticed that this doesn't get triggered in my logs: https://github.com/jenkinsci/azure-vm-agents-plugin/blob/master/src/main/java/com/microsoft/azure/vmagent/AzureVMAgentCleanUpTask.java#L345-L349, but still shows as 0 for deletion.

timja commented 2 years ago

Yes there's a number of those bugs around, the code isn't written write for proper interpolation in quite a few places

shanep2300 commented 1 year ago

@timja Figured out the RG error. There was a cloud config using the wrong SP.

@ryanberger-az The SP has Contributor role at the subscription level.

After these changes and no update on the plugin on our end, it looks like it has correctly fixed itself. Azure portal is now showing the VMs that are currently running and all others are deleted correctly.

Can't reproduce this issue so I'm going to close it.