search

jenkinsci / amazon-ecs-plugin

Amazon EC2 Container Service Plugin for Jenkins
https://plugins.jenkins.io/amazon-ecs
MIT License
192 stars 226 forks source link

Upgrading to v. 1.46 causes ecs jobs to not start. #291

Closed jjardina closed 1 year ago

jjardina commented 1 year ago

Jenkins and plugins versions report

Jenkins: 2.361.3 OS: Linux - 5.15.0-1022-aws

ace-editor:1.1 amazon-ecs:1.41 ansicolor:1.0.2 ant:481.v7b_09e538fcca antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifactory:3.17.3 atlassian-jira-software-cloud:2.0.7 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd blueocean:1.25.8 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.8 blueocean-commons:1.25.8 blueocean-config:1.25.8 blueocean-core-js:1.25.8 blueocean-dashboard:1.25.8 blueocean-display-url:2.4.1 blueocean-events:1.25.8 blueocean-git-pipeline:1.25.8 blueocean-github-pipeline:1.25.8 blueocean-i18n:1.25.8 blueocean-jira:1.25.8 blueocean-jwt:1.25.8 blueocean-personalization:1.25.8 blueocean-pipeline-api-impl:1.25.8 blueocean-pipeline-editor:1.25.8 blueocean-pipeline-scm-api:1.25.8 blueocean-rest:1.25.8 blueocean-rest-impl:1.25.8 blueocean-web:1.25.8 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc build-discarder:139.v05696a_7fe240 build-monitor-plugin:1.13+build.202205140447 build-name-setter:2.2.0 build-timeout:1.24 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.0 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.758.vfd75d09eea_a_1 command-launcher:90.v669d7ccb7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b commons-text-api:1.10.0-27.vb_fa_3896786a_7 config-file-provider:3.11.1 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 docker-commons:1.21 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.10 docker-workflow:528.v7c193a_0b_e67c durable-task:501.ve5d4fc08b0be echarts-api:5.4.0-1 email-ext:2.92 favorite:2.4.1 font-awesome-api:6.2.0-3 git:4.13.0 git-client:3.13.0 git-parameter:0.9.18 github:1.36.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 gitlab-api:5.0.1-78.v47a_45b_9f78b_7 gitlab-branch-source:642.v9ed86b_b_54384 gitlab-logo:1.1.0 gitlab-plugin:1.5.36 gradle:2.1 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:116.vf8f487400980 ionicons-api:31.v4757b_6987003 ivy:2.2 jackson2-api:2.13.4.20221013-295.v8e29ea_354141 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.7-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.25.8 jersey2-api:2.37-1 jira:3.8 jira-steps:2.0.141.vd0c6e6dc83f0 jjwt-api:0.11.5-77.v646c772fddb_0 jobConfigHistory:1176.v1b_4290db_41a_5 jquery:1.12.4-1 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1159.v0b_396e1e07dd ldap:2.12 log-parser:2.3.0 mailer:438.v02c7f0a_12fa_4 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.5 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.20 metrics:4.2.10-389.v93143621b_050 mina-sshd-api-common:2.9.1-44.v476733c11f82 mina-sshd-api-core:2.9.1-44.v476733c11f82 momentjs:1.1.1 next-build-number:1.8 npm-yarn-wrapper-steps:0.4.0 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 pipeline-build-step:2.18 pipeline-github-lib:38.v445716eaedda pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:613.v9c41a_160233f pipeline-input-step:456.vd8a_957db_5b_e9 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.27 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.27 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.18.0 popper2-api:2.11.6-2 pubsub-light:1.17 remote-file:1.23 resource-disposer:0.20 role-strategy:562.v44e9a_e828d0e scm-api:621.vda_a_b_055e58f7 script-security:1189.vb_a_b_7c8fd5fde snakeyaml-api:1.33-90.v80dcb_3814d35 sonar:2.14 sse-gateway:1.26 ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.249.v2dc2ea_416e33 structs:324.va_f5d6774f3a_d subversion:2.16.0 sumologic-publisher:2.2.1 timestamper:1.21 token-macro:308.v4f2b_ed62b_b_16 trilead-api:2.72.v2a_3236754f73 variant:59.vf075fe829ccb workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:3536.vb_8a_6628079d5 workflow-durable-task-step:1210.va_1e5d77e122b workflow-job:1254.v3f64639b_11dd workflow-multibranch:716.vc692ae52371b workflow-scm-step:400.v6b_89a1317c9a workflow-step-api:639.v6eca_cd8c04aa workflow-support:839.v35e2736cfd5c ws-cleanup:0.43

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu Server 20.04

Reproduction steps

  1. Have version 1.41 ecs/fargate plug-in
  2. Upgrade to 1.46 ecs/fargate plug-in
  3. ECS jobs hang, never start
  4. Look at system log and see ecs jobs trying to start but get error and restart
  5. Downgrade plugin to 1.41
  6. ECS jobs run normally

Expected Results

ecs plugin v. 1.41: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-west-2:970463182732:task-definition/ecs-cluster-jenkins-inbound-agent:122,ContainerDefinitions: [{Name: ecs-cluster-jenkins-inbound-agent,Image: valisecurity.jfrog.io/jenkins-agent/jenkins-build:0.3,RepositoryCredentials: {CredentialsParameter: arn:aws:secretsmanager:us-west-2:970463182732:secret:jfrog_token-aZxQlR},Cpu: 256,MemoryReservation: 1024,Links: [],PortMappings: [],Essential: true,EntryPoint: [],Command: [],Environment: [],EnvironmentFiles: [],MountPoints: [],VolumesFrom: [],Secrets: [],DependsOn: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],SystemControls: [],ResourceRequirements: [],}],Family: ecs-cluster-jenkins-inbound-agent,ExecutionRoleArn: arn:aws:iam::970463182732:role/ecsTaskExecutionRole,NetworkMode: host,Revision: 122,Volumes: [],Status: ACTIVE,RequiresAttributes: [{Name: ecs.capability.private-registry-authentication.secretsmanager,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}],PlacementConstraints: [],Compatibilities: [EXTERNAL, EC2],RequiresCompatibilities: [],InferenceAccelerators: [],RegisteredAt: Fri Nov 11 15:20:54 UTC 2022,RegisteredBy: arn:aws:iam::970463182732:user/jason_dev,} Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask [ecs-cluster-ecs-5pcbx]: Starting agent with task definition arn:aws:ecs:us-west-2:970463182732:task-definition/ecs-cluster-jenkins-inbound-agent:122} Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask [ecs-cluster-ecs-5pcbx]: Agent started with task arn : arn:aws:ecs:us-west-2:970463182732:task/ecs-cluster/6dc72a658044468ea74f56ebef3ac14f Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchECSTask [ecs-cluster-ecs-5pcbx]: TaskArn: arn:aws:ecs:us-west-2:970463182732:task/ecs-cluster/6dc72a658044468ea74f56ebef3ac14f Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchECSTask [ecs-cluster-ecs-5pcbx]: TaskDefinitionArn: arn:aws:ecs:us-west-2:970463182732:task-definition/ecs-cluster-jenkins-inbound-agent:122 Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchECSTask [ecs-cluster-ecs-5pcbx]: ClusterArn: arn:aws:ecs:us-west-2:970463182732:cluster/ecs-cluster Nov 11, 2022 3:20:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchECSTask [ecs-cluster-ecs-5pcbx]: ContainerInstanceArn: arn:aws:ecs:us-west-2:970463182732:container-instance/ecs-cluster/a52edfe56b0447018fdf3314ca63069d Nov 11, 2022 3:21:53 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launchECSTask [ecs-cluster-ecs-5pcbx]: Task started, waiting for agent to become online Nov 11, 2022 3:21:53 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher waitForAgent [ecs-cluster-ecs-5pcbx]: Waiting for agent to connect Nov 11, 2022 3:21:53 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run Accepted JNLP4-connect connection #1 from ip-172-16-1-42.us-west-2.compute.internal/172.16.1.42:46598 Nov 11, 2022 3:21:54 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher waitForAgent [ecs-cluster-ecs-5pcbx]: Waiting for agent to connect Nov 11, 2022 3:21:55 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher waitForAgent [ecs-cluster-ecs-5pcbx]: Agent connected Nov 11, 2022 3:21:55 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSComputer taskAccepted [ecs-cluster-ecs-5pcbx]: JobName: part of Testing » ECS-Pipeline-Test #28 Nov 11, 2022 3:21:55 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSComputer taskAccepted [ecs-cluster-ecs-5pcbx]: JobUrl: job/Testing/job/ECS-Pipeline-Test/28/ Nov 11, 2022 3:22:00 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSSlave _terminate [ecs-cluster-ecs-5pcbx]: Stopping: TaskArn arn:aws:ecs:us-west-2:970463182732:task/ecs-cluster/6dc72a658044468ea74f56ebef3ac14f, ClusterArn arn:aws:ecs:us-west-2:970463182732:cluster/ecs-cluster Nov 11, 2022 3:22:00 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSService stopTask Delete ECS agent task: arn:aws:ecs:us-west-2:970463182732:task/ecs-cluster/6dc72a658044468ea74f56ebef3ac14f Nov 11, 2022 3:22:00 PM INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Computer.threadPoolForRemoting [#7] for ecs-cluster-ecs-5pcbx terminated: java.nio.channels.ClosedChannelException Nov 11, 2022 3:22:26 PM INFO hudson.model.AsyncPeriodicWork lambda$doRun$1

Actual Results

ecs plugin v. 1.46 Nov 11, 2022 3:26:02 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Asked to provision 1 agent(s) for: ecs Nov 11, 2022 3:26:02 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Will provision ecs-cluster-ecs-txkbg, for label: ecs Nov 11, 2022 3:26:02 PM SEVERE hudson.triggers.SafeTimerTask run Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@1def51e1 failed java.lang.IllegalArgumentException at hudson.slaves.NodeProvisioner$PlannedNode.(NodeProvisioner.java:102) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.provision(ECSCloud.java:283) at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:65) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Nov 11, 2022 3:26:12 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Asked to provision 1 agent(s) for: ecs Nov 11, 2022 3:26:12 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Will provision ecs-cluster-ecs-s39z7, for label: ecs Nov 11, 2022 3:26:12 PM SEVERE hudson.triggers.SafeTimerTask run Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@1def51e1 failed java.lang.IllegalArgumentException at hudson.slaves.NodeProvisioner$PlannedNode.(NodeProvisioner.java:102) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.provision(ECSCloud.java:283) at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:65) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Nov 11, 2022 3:26:22 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Asked to provision 1 agent(s) for: ecs Nov 11, 2022 3:26:22 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Will provision ecs-cluster-ecs-zp8w2, for label: ecs Nov 11, 2022 3:26:22 PM SEVERE hudson.triggers.SafeTimerTask run Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@1def51e1 failed java.lang.IllegalArgumentException at hudson.slaves.NodeProvisioner$PlannedNode.(NodeProvisioner.java:102) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.provision(ECSCloud.java:283) at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:65) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Nov 11, 2022 3:26:32 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Asked to provision 1 agent(s) for: ecs Nov 11, 2022 3:26:32 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSCloud provision Will provision ecs-cluster-ecs-ms80q, for label: ecs Nov 11, 2022 3:26:32 PM SEVERE hudson.triggers.SafeTimerTask run Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@1def51e1 failed java.lang.IllegalArgumentException at hudson.slaves.NodeProvisioner$PlannedNode.(NodeProvisioner.java:102) at com.cloudbees.jenkins.plugins.amazonecs.ECSCloud.provision(ECSCloud.java:283) at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:65) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

No response

btomblin commented 1 year ago

Same issue, was able to work around by setting number of executors using CaC plugin:

 clouds:
  - ecs:
      numExecutors: 1

Can also update "Number of executors per agent" in the console.

jjardina commented 1 year ago

Same issue, was able to work around by setting number of executors using CaC plugin:

 clouds:
  - ecs:
      numExecutors: 1

Can also update "Number of executors per agent" in the console.

This worked for me. Thank you @btomblin The default value in the console is 0, which causes all ecs jobs to fail. It appears that v. 1.46 honors this value where v. 1.41 did not.

Maybe the default should be set to 1 at plugin install and allow only a value of 1 or greater -or- make that option mandatory and document it as it is hidden under Advanced button.

image

tgray-hrg commented 1 year ago

I tried to change the Number of executors per agent to 1 and my pending tasks still wouldn't start up. I had to revert back. I am using Jenkins 2.332.2

munjalt commented 1 year ago

@tgray-hrg there are other issues in the plugin. Apparently there are other values that the plugin checks to make sure they aren't empty.

I suggest you check the stack trace. I just fixed the issue I was having by adding a simple tag value. The number of executors were set and I still got the exception.

java.lang.NullPointerException at com.cloudbees.jenkins.plugins.amazonecs.ECSService.registerTemplate(ECSService.java:310) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launchECSTask(ECSLauncher.java:148) at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:110) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

If you look at the code ECSService.java at line 310 this is what it's looking for https://github.com/jenkinsci/amazon-ecs-plugin/blob/5fda2a6d57cc49c88036f957fd4a3ccc8b407175/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java#L310

for (ECSTaskTemplate.Tag tag: template.getTags()) { taskDefinitionTags.add(new Tag().withKey(tag.name).withValue(tag.value)); }

btomblin commented 1 year ago

@munjalt - I noticed that issue occurred when updating an existing container definition under:

Configure Clouds -> Amazon EC2 Container Service Cloud

The work around was creating a dummy tag:

image

@tgray-hrg - What error message were you seeing in the logs?

munjalt commented 1 year ago

@btomblin thanks. Yes, I believe that the issue happens when you try to update an existing ecs cloud definition or add another from the console. In my case it was me trying to add a new cloud definition from console. All settings were exactly the same in JCASC output. So, to anyone having an issue I suggest just check the stack trace from the errors and trace which settings it's complaining about. Jenkins logs are available at JENKINS_URL/manage/log/all. If you want help just paste them here and we can try to help.

tgray-hrg commented 1 year ago

@munjalt - I noticed that issue occurred when updating an existing container definition under:

Configure Clouds -> Amazon EC2 Container Service Cloud

The work around was creating a dummy tag:

image

@tgray-hrg - What error message were you seeing in the logs?

Adding the dummy tags worked. Thank you

pfrickcol commented 1 year ago

I encountered this same issue. Resolution was to set executors on the agent to 1 and to add a tag to each definition. I had one that was still erroring out 

[1_0_1024_2048-snm1v]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[1_0_1024_2048-snm1v]
com.amazonaws.services.ecs.model.InvalidParameterException: Task definition does not support launch_type FARGATE. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 890a43fc-f462-400c-85c1-a33f78a49b24; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.ecs.AmazonECSClient.doInvoke(AmazonECSClient.java:4691)
    at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:4658)
    at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:4647)
    at com.amazonaws.services.ecs.AmazonECSClient.executeRunTask(AmazonECSClient.java:3295)
    at com.amazonaws.services.ecs.AmazonECSClient.runTask(AmazonECSClient.java:3264)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:496)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:213)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launchECSTask(ECSLauncher.java:149)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:110)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

To resolve this I deleted the task definition saved it then re-created the task definition and it worked.

BongoEADGC6 commented 1 year ago

Same issue, was able to work around by setting number of executors using CaC plugin:

 clouds:
  - ecs:
      numExecutors: 1

Can also update "Number of executors per agent" in the console.

This worked for me. Thank you @btomblin The default value in the console is 0, which causes all ecs jobs to fail. It appears that v. 1.46 honors this value where v. 1.41 did not.

Maybe the default should be set to 1 at plugin install and allow only a value of 1 or greater -or- make that option mandatory and document it as it is hidden under Advanced button.

image

This was the confirmed solution for our instances as well.

hemp commented 1 year ago

We hit this issue today with 1.46. Worth noting that this JCasC config above was useful to fix the java.lang.IllegalArgumentException.

Then we then hit the IAM permission requiring ecs:ListTagsForResource https://github.com/jenkinsci/amazon-ecs-plugin/issues/290#issuecomment-1310967091

westinn commented 1 year ago

Just wanted to report that I am facing the same issue with the latest version (1.46). I tried following the suggestion in the linked docs in the error log by setting hudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID to true, but with 1.46 it did not fix anything.

Downgrading to 1.41 solved the problem.

Aub-C commented 1 year ago

I am having the same issue with 1.46. I tried updating the numExecutors: 1, however, this did not work for me. I had to revert back to 1.41, which resolved my issue. My tasks are running again.

qalinn commented 1 year ago

Any updates on this issue? Also, I have the same problem.

Stericson commented 1 year ago

I've pushed a fix for the number of executors issue. Hope to get a new version out before EOY.

Stericson commented 1 year ago

1.47 is out. Thanks for the patience!