Use :latest task definition instead of registering a new one every time.

pguartes-endava commented 2 years ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.332.3 OS: Linux - 4.14.275-207.503.amzn2.x86_64 --- ace-editor:1.1 amazon-ecs:1.40 ant:1.13 antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-1.0 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df bitbucket:223.vd12f2bca5430 blueocean:1.25.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.25.3 blueocean-commons:1.25.3 blueocean-config:1.25.3 blueocean-core-js:1.25.3 blueocean-dashboard:1.25.3 blueocean-display-url:2.4.1 blueocean-events:1.25.3 blueocean-git-pipeline:1.25.3 blueocean-github-pipeline:1.25.3 blueocean-i18n:1.25.3 blueocean-jwt:1.25.3 blueocean-personalization:1.25.3 blueocean-pipeline-api-impl:1.25.3 blueocean-pipeline-editor:1.25.3 blueocean-pipeline-scm-api:1.25.3 blueocean-rest:1.25.3 blueocean-rest-impl:1.25.3 blueocean-web:1.25.3 bootstrap4-api:4.6.0-3 bootstrap5-api:5.1.3-6 bouncycastle-api:2.25 branch-api:2.7.0 build-timeout:1.20 caffeine-api:2.9.2-29.v717aac953ff3 checks-api:1.7.2 cloudbees-bitbucket-branch-source:757.vddedc5f2589a_ cloudbees-folder:6.708.ve61636eb_65a_5 command-launcher:1.6 configuration-as-code:1414.v878271fc496f credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:1.27.1 display-url-api:2.3.5 durable-task:493.v195aefbb0ff2 echarts-api:5.3.0-2 email-ext:2.87 favorite:2.4.1 font-awesome-api:6.0.0-1 git:4.10.3 git-client:3.11.0 git-parameter:0.9.16 git-server:1.10 github:1.34.3 github-api:1.301-378.v9807bd746da5 github-branch-source:2.11.4 gradle:1.38 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-1.0 htmlpublisher:1.29 jackson2-api:2.13.1-246.va8a9f3eaf46a javax-activation-api:1.2.0-2 javax-mail-api:1.6.2-5 jaxb:2.3.0.1 jdk-tool:1.5 jenkins-design-language:1.25.3 jjwt-api:0.11.2-9.c8b45b8bb173 jnr-posix-api:3.1.7-2 jquery:1.12.4-1 jquery3-api:3.6.0-2 jsch:0.1.55.2 junit:1.54 ldap:2.8 locale:144.v1a_998824ddb_3 lockable-resources:2.14 mailer:408.vd726a_1130320 matrix-auth:3.1 matrix-project:1.20 mercurial:2.16 momentjs:1.1.1 okhttp-api:4.9.3-105.vb96869f8ac3a pam-auth:1.7 pipeline-aws:1.43 pipeline-build-step:2.16 pipeline-github-lib:36.v4c01db_ca_ed16 pipeline-graph-analysis:188.v3a01e7973f2c pipeline-input-step:446.vf27b_0b_83500e pipeline-milestone-step:100.v60a_03cd446e1 pipeline-model-api:1.9.3 pipeline-model-definition:1.9.3 pipeline-model-extensions:1.9.3 pipeline-rest-api:2.23 pipeline-stage-step:291.vf0a8a7aeeb50 pipeline-stage-tags-metadata:1.9.3 pipeline-stage-view:2.23 plain-credentials:1.8 plugin-util-api:2.14.0 popper-api:1.16.1-2 popper2-api:2.11.2-1 pubsub-light:1.16 resource-disposer:0.17 role-strategy:3.2.0 scm-api:595.vd5a_df5eb_0e39 script-security:1138.v8e727069a_025 snakeyaml-api:1.29.1 sonar:2.14 sse-gateway:1.25 ssh-credentials:1.19 ssh-slaves:1.806.v2253cedd3295 sshd:3.1.0 structs:308.v852b473a2b8c timestamper:1.17 token-macro:280.v97a_82642793c trilead-api:1.0.13 variant:1.4 workflow-aggregator:2.7 workflow-api:1138.v619fd5201b_2f workflow-basic-steps:2.24 workflow-cps:2660.vb_c0412dc4e6d workflow-cps-global-lib:566.vd0a_a_3334a_555 workflow-durable-task-step:1121.va_65b_d2701486 workflow-job:1145.v7f2433caa07f workflow-multibranch:711.vdfef37cda_816 workflow-scm-step:2.13 workflow-step-api:622.vb_8e7c15b_c95a_ workflow-support:813.vb_d7c3d2984a_0 ws-cleanup:0.40 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Linux - 4.14.275-207.503.amzn2.x86_64

Reproduction steps

Configure ECS template with a template name

Expected Results

Plugin reuse the same task definition on ECS

Actual Results

The plugin creates one task definition per Job, which starts to increase the number of task definitions for the aws account. Currently we have thousands of revisions of the same task definition.

Anything else?

We tried Task Definition Override option but it seems to work for that purpose but Jobs don't run in parallel. Is there another way?

cmlara commented 2 years ago

For those who come across this in the future, the following log lines are most relevant to determine why a task definition is being created

INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on container definition: true
INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on volumes: true
INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on task role: true
INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on execution role: true
INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on network mode: false

In my deployment the Match on network was always false when I utilized "default" for the "Network mode" (this is for a Fargate deployment)

Looking at https://github.com/jenkinsci/amazon-ecs-plugin/blob/a3cffcb9d7f8b2aa79e758a0422e6535311e3ad0/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSService.java#L281 I do not believe that the test could ever return a TRUE result when using "default" as I do not believe getNetworkMode() can ever return NULL for an existing task definition.

Setting the "Network mode" to the appropriate network type instead of default solved the issue in my deployment. This may or may not be the cause for the OP.

Sebasti1492 commented 2 years ago

Same issue. The task definition is created every time for each job though there is no change in the definition. Log shows container definition no match however the container definition is still the same for the agent versions. As per Jenkins logs as below:

2022-07-15 18:46:31.201+0000 [id=39]    INFO    c.c.j.plugins.amazonecs.ECSCloud#provision: Asked to provision 1 agent(s) for: aloc-linux-agent
2022-07-15 18:46:31.202+0000 [id=39]    INFO    c.c.j.plugins.amazonecs.ECSCloud#provision: Will provision app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q, for label: aloc-linux-agent
2022-07-15 18:46:41.250+0000 [id=33]    INFO    hudson.slaves.NodeProvisioner#update: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q provisioning successfully completed. We have now 19 computer(s)
2022-07-15 18:46:41.276+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSService#findTaskDefinition: No existing task definition found for family or ARN: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent
2022-07-15 18:46:41.394+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1,ContainerDefinitions: [{Name: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,Image: <accntid>.dkr.ecr.us-east-1.amazonaws.com/app-uai2005906-trs-jenkins-nonprod-linux-agent:RELEASE,Cpu: 2048,MemoryReservation: 4096,Links: [],PortMappings: [],Essential: true,EntryPoint: [/jenkins_exec/jenkins-agent],Command: [],Environment: [{Name: JENKINS_AGENT_WORKDIR,Value: /jenkins_home/jenkins}, {Name: JENKINS_URL,Value: https://jenkins-dev.xxxxx.xxx.com}],EnvironmentFiles: [],MountPoints: [{SourceVolume: jenkins-efs,ContainerPath: /jenkins_home,ReadOnly: false}],VolumesFrom: [],Secrets: [],DependsOn: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],LogConfiguration: {LogDriver: awslogs,Options: {awslogs-group=/aws/ecs/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-ecs-task, awslogs-region=us-east-1, awslogs-stream-prefix=ecs-task},SecretOptions: []},SystemControls: [],ResourceRequirements: [],}],Family: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,TaskRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-linux-agent-ecs-task-role,ExecutionRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-ecs-task-exec-role,NetworkMode: awsvpc,Revision: 1,Volumes: [{Name: jenkins-efs,EfsVolumeConfiguration: {FileSystemId: fs-0d90702807251532d,RootDirectory: /,TransitEncryption: ENABLED,AuthorizationConfig: {AccessPointId: fsap-0c57d79981273aeb6,Iam: ENABLED}},}],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.logging-driver.awslogs,}, {Name: ecs.capability.execution-role-awslogs,}, {Name: ecs.capability.efsAuth,}, {Name: com.amazonaws.ecs.capability.ecr-auth,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.19,}, {Name: ecs.capability.efs,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}, {Name: com.amazonaws.ecs.capability.task-iam-role,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.25,}, {Name: ecs.capability.execution-role-ecr-pull,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}, {Name: ecs.capability.task-eni,}],PlacementConstraints: [],Compatibilities: [EC2, FARGATE],RuntimePlatform: {CpuArchitecture: X86_64,OperatingSystemFamily: LINUX},RequiresCompatibilities: [FARGATE],Cpu: 2048,Memory: 4096,InferenceAccelerators: [],RegisteredAt: Fri Jul 15 18:46:41 UTC 2022,RegisteredBy: arn:aws:sts::<accntid>:assumed-role/app-uai2005906-trs-jenkins-nonprod-master-ecs-task-role/c42850fa9db249d7828d31e6faf649de,}
2022-07-15 18:46:41.394+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Starting agent with task definition arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1}
2022-07-15 18:46:42.076+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Agent started with task arn : arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/6672adb7700040d783f7642fd4075cdc
2022-07-15 18:46:42.076+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: TaskArn: arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/6672adb7700040d783f7642fd4075cdc
2022-07-15 18:46:42.076+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:1
2022-07-15 18:46:42.076+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: ClusterArn: arn:aws:ecs:us-east-1:<accntid>:cluster/app-uai2005906-trs-jenkins-nonprod-ecs-cluster
2022-07-15 18:46:42.076+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: ContainerInstanceArn: null
2022-07-15 18:46:42.153+0000 [id=619]   INFO    hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$1: Started Update IdP Metadata from URL PeriodicWork
2022-07-15 18:46:42.153+0000 [id=619]   INFO    hudson.model.AsyncAperiodicWork#lambda$doAperiodicRun$1: Finished Update IdP Metadata from URL PeriodicWork. 0 ms
2022-07-15 18:48:13.310+0000 [id=41]    INFO    o.j.p.P.u.DropCachePeriodicWork#doRun: begin schedule clean...
2022-07-15 18:48:13.310+0000 [id=41]    INFO    o.j.p.P.u.DropCachePeriodicWork#doRun: end schedule clean...
2022-07-15 18:49:08.817+0000 [id=682]   INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #9 from /x.x.x.x:34922
2022-07-15 18:49:21.201+0000 [id=40]    INFO    c.c.j.plugins.amazonecs.ECSCloud#provision: Asked to provision 1 agent(s) for: aloc-linux-agent
2022-07-15 18:49:21.202+0000 [id=40]    INFO    c.c.j.plugins.amazonecs.ECSCloud#provision: Will provision app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h, for label: aloc-linux-agent
2022-07-15 18:49:31.244+0000 [id=42]    INFO    hudson.slaves.NodeProvisioner#update: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h provisioning successfully completed. We have now 20 computer(s)
2022-07-15 18:49:31.270+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on container definition: false
2022-07-15 18:49:31.270+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on volumes: true
2022-07-15 18:49:31.270+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on task role: true
2022-07-15 18:49:31.271+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on execution role: true
2022-07-15 18:49:31.271+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Match on network mode: true
2022-07-15 18:49:31.397+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSService#registerTemplate: Created Task Definition: {TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2,ContainerDefinitions: [{Name: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,Image: <accntid>.dkr.ecr.us-east-1.amazonaws.com/app-uai2005906-trs-jenkins-nonprod-linux-agent:RELEASE,Cpu: 2048,MemoryReservation: 4096,Links: [],PortMappings: [],Essential: true,EntryPoint: [/jenkins_exec/jenkins-agent],Command: [],Environment: [{Name: JENKINS_AGENT_WORKDIR,Value: /jenkins_home/jenkins}, {Name: JENKINS_URL,Value: https://jenkins-dev.xxxxx.xxx.com}],EnvironmentFiles: [],MountPoints: [{SourceVolume: jenkins-efs,ContainerPath: /jenkins_home,ReadOnly: false}],VolumesFrom: [],Secrets: [],DependsOn: [],Privileged: false,DnsServers: [],DnsSearchDomains: [],ExtraHosts: [],DockerSecurityOptions: [],Ulimits: [],LogConfiguration: {LogDriver: awslogs,Options: {awslogs-group=/aws/ecs/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-ecs-task, awslogs-region=us-east-1, awslogs-stream-prefix=ecs-task},SecretOptions: []},SystemControls: [],ResourceRequirements: [],}],Family: app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent,TaskRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-linux-agent-ecs-task-role,ExecutionRoleArn: arn:aws:iam::<accntid>:role/app/uai2005906/app-uai2005906-trs-jenkins-nonprod-ecs-task-exec-role,NetworkMode: awsvpc,Revision: 2,Volumes: [{Name: jenkins-efs,EfsVolumeConfiguration: {FileSystemId: fs-0d90702807251532d,RootDirectory: /,TransitEncryption: ENABLED,AuthorizationConfig: {AccessPointId: fsap-0c57d79981273aeb6,Iam: ENABLED}},}],Status: ACTIVE,RequiresAttributes: [{Name: com.amazonaws.ecs.capability.logging-driver.awslogs,}, {Name: ecs.capability.execution-role-awslogs,}, {Name: ecs.capability.efsAuth,}, {Name: com.amazonaws.ecs.capability.ecr-auth,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.19,}, {Name: ecs.capability.efs,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.21,}, {Name: com.amazonaws.ecs.capability.task-iam-role,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.25,}, {Name: ecs.capability.execution-role-ecr-pull,}, {Name: com.amazonaws.ecs.capability.docker-remote-api.1.18,}, {Name: ecs.capability.task-eni,}],PlacementConstraints: [],Compatibilities: [EC2, FARGATE],RuntimePlatform: {CpuArchitecture: X86_64,OperatingSystemFamily: LINUX},RequiresCompatibilities: [FARGATE],Cpu: 2048,Memory: 4096,InferenceAccelerators: [],RegisteredAt: Fri Jul 15 18:49:31 UTC 2022,RegisteredBy: arn:aws:sts::<accntid>:assumed-role/app-uai2005906-trs-jenkins-nonprod-master-ecs-task-role/c42850fa9db249d7828d31e6faf649de,}
2022-07-15 18:49:31.397+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: Starting agent with task definition arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2}
2022-07-15 18:49:32.206+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#runECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: Agent started with task arn : arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/e820fea89540483199cf135c67fc787e
2022-07-15 18:49:32.207+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: TaskArn: arn:aws:ecs:us-east-1:<accntid>:task/app-uai2005906-trs-jenkins-nonprod-ecs-cluster/e820fea89540483199cf135c67fc787e
2022-07-15 18:49:32.207+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: TaskDefinitionArn: arn:aws:ecs:us-east-1:<accntid>:task-definition/app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent:2
2022-07-15 18:49:32.207+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: ClusterArn: arn:aws:ecs:us-east-1:<accntid>:cluster/app-uai2005906-trs-jenkins-nonprod-ecs-cluster
2022-07-15 18:49:32.207+0000 [id=694]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-kwl4h]: ContainerInstanceArn: null
2022-07-15 18:49:42.200+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#launchECSTask: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Task started, waiting for agent to become online
2022-07-15 18:49:42.201+0000 [id=546]   INFO    c.c.j.p.amazonecs.ECSLauncher#waitForAgent: [app-uai2005906-trs-jenkins-nonprod-aloc-linux-agent-crm3q]: Agent connected

helloPiers commented 1 year ago

I notice two other cases where the matching fails, falsely: 1) If the container definition contains environment variables, these are not received in any defined order from ECS API, so most often the container definition is deemed different, even though it's the same env pairs, just in a different order 2) The automatic tags "jenkins.label" and "jenkins.templatename" are not added to the "wanted" list of tags before comparing with the previously registered template's tags (which does contain those), so it always thinks the tags are different. (It's likely the same sort order thing applies here, too.)

helloPiers commented 1 year ago

Actually, checking the code, the second of the above cases isn't actually effective, because although the tags are checked and difference (incorrectly) reported, no further reference to that difference is made in the code afterwards.

(templateTagsMatchesExistingTags is set, logged, but not tested in the if later.)

pluralistix commented 10 months ago

Hey,

we have created https://github.com/aws/aws-sdk-java/issues/3043 for it and also opened a support case with AWS.

First saw https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html "Revisions per task definition family: 1,000,000" and thought....Pff...And then

❯ aws ecs list-task-definitions --family-prefix jenkins-build-cluster-xxx --status ACTIVE --max-items 2147483647 | jq -r '.taskDefinitionArns|length'
725010

Oh... 😄 (or 😱 - depends)...

BR

jenkinsci / amazon-ecs-plugin