InvalidParameterException: Override argument cannot be null, Creating Task in cluster null

jlujan-invitae commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.387.3 OS: Linux - 5.10.179-168.710.amzn2.x86_64 Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.18.0 ace-editor:1.1 amazon-ecs:1.48 ant:487.vd79d090d4ea_e antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 artifact-manager-s3:822.vf129d4836c31 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:108.v47b_fd43dfec6 aws-java-sdk:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-cloudformation:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-codebuild:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-ec2:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-ecr:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-ecs:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-efs:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-elasticbeanstalk:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-iam:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-kinesis:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-logs:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-minimal:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-sns:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-sqs:1.12.481-392.v8b_291cfcda_09 aws-java-sdk-ssm:1.12.481-392.v8b_291cfcda_09 aws-secrets-manager-credentials-provider:1.202.ve0ec0c17611c blueocean:1.27.4 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.4 blueocean-commons:1.27.4 blueocean-config:1.27.4 blueocean-core-js:1.27.4 blueocean-dashboard:1.27.4 blueocean-display-url:2.4.2 blueocean-events:1.27.4 blueocean-git-pipeline:1.27.4 blueocean-github-pipeline:1.27.4 blueocean-i18n:1.27.4 blueocean-jwt:1.27.4 blueocean-personalization:1.27.4 blueocean-pipeline-api-impl:1.27.4 blueocean-pipeline-editor:1.27.4 blueocean-pipeline-scm-api:1.27.4 blueocean-rest:1.27.4 blueocean-rest-impl:1.27.4 blueocean-web:1.27.4 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.0-1 bouncycastle-api:2.28 branch-api:2.1109.vdf225489a_16d build-timeout:1.31 caffeine-api:3.1.6-115.vb_8b_b_328e59d8 checks-api:2.0.0 cloudbees-bitbucket-branch-source:809.vc1d904b_30426 cloudbees-folder:6.815.v0dd5a_cb_40e0e cobertura:1.17 code-coverage-api:4.7.0 command-launcher:100.v2f6722292ee8 commons-httpclient3-api:3.1-3 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ config-file-provider:938.ve2b_8a_591c596 configuration-as-code:1647.ve39ca_b_829b_42 credentials:1254.vb_96f366e7b_a_d credentials-binding:604.vb_64480b_c56ca_ data-tables-api:1.13.4-2 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c docker-workflow:563.vd5d2e5c4007f durable-task:507.v050055d0cb_dd echarts-api:5.4.0-5 favorite:2.4.2 font-awesome-api:6.4.0-1 forensics-api:2.3.0 generic-webhook-trigger:1.86.3 git:5.1.0 git-client:4.4.0 git-server:99.va_0826a_b_cdfa_d github:1.37.1 github-api:1.314-431.v78d72a_3fe4c3 github-branch-source:1728.v859147241f49 github-checks:545.v79a_a_68b_ca_682 gradle:2.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e ivy:2.5 jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:233.vdc1a_ec702cff javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:66.vd8fa_64ee91b_d jenkins-design-language:1.27.4 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.84 jquery3-api:3.7.0-1 jsch:0.2.8-65.v052c39de79b_2 junit:1207.va_09d5100410f labelled-steps:1.0 lockable-resources:1166.v7ef984f72fe2 mailer:457.v3f72cb_e015e5 matrix-auth:3.1.8 matrix-project:789.v57a_725b_63c79 maven-plugin:3.22 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ nexus-artifact-uploader:2.14 okhttp-api:4.11.0-145.vcb_8de402ef81 parameterized-scheduler:1.2 pipeline-aws:1.43 pipeline-build-step:496.v2449a_9a_221f2 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:468.va_5db_051498a_4 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2141.v5402e818a_779 pipeline-model-definition:2.2141.v5402e818a_779 pipeline-model-extensions:2.2141.v5402e818a_779 pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2141.v5402e818a_779 pipeline-stage-view:2.32 pipeline-utility-steps:2.15.4 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-7 pubsub-light:1.17 resource-disposer:0.22 role-strategy:633.v836e5b_3e80a_5 saml:4.418.vdfa_7489a_b_a_2d scm-api:676.v886669a_199a_a_ script-security:1251.vfe552ed55f8d slack:664.vc9a_90f8b_c24a_ snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sse-gateway:1.26 ssh-credentials:305.v8f4381501156 sshd:3.303.vefc7119b_ec23 structs:324.va_f5d6774f3a_d timestamper:1.25 token-macro:359.vb_cde11682e0c trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1215.v2b_ee3e1b_dd39 workflow-basic-steps:1017.vb_45b_302f0cea_ workflow-cps:3691.v28b_14c465a_b_b_ workflow-durable-task-step:1247.v7f9dfea_b_4fd0 workflow-job:1308.v58d48a_763b_31 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

Create new Cloud in Jenkins console Configure Cloud for ECS cluster Add Template to Cloud Create Job with scripted pipeline that utilizes ECSTaskTemplate to inherit from the cloud template Run the Job

Job

def dynamic_label = "${JOB_NAME}${currentBuild.number}".replaceAll('[^a-zA-Z0-9]', '').toLowerCase()

ecsTaskTemplate(
    cloud: 'on-prem-agent-cluster',
    label: dynamic_label,
    name: dynamic_label,
    inheritFrom: 'on-prem-agent-template',
    image: '<ECR Registry>/ruo-analysis-agent:latest',
    memoryReservation: 4096,
    cpu: "1024",
    overrides: []
)
{
  node(dynamic_label)
  ...
  }

Expected Results

Task is created in ECS.

Actual Results

Jun 17, 2023 9:48:12 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy

Started provisioning on-prem-agent-cluster-onpremtest68-htgt1 from on-prem-agent-cluster with 1 executors. Remaining excess workload: 0

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher

ECS: Launching agent

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher

[on-prem-agent-cluster-onpremtest68-htgt1]: Creating Task in cluster null

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy

Received StrategyState{label=onpremtest68, snapshot=LoadStatisticsSnapshot{definedExecutors=1, onlineExecutors=0, connectingExecutors=1, busyExecutors=0, idleExecutors=0, availableExecutors=0, queueLength=1}, plannedCapacitySnapshot=0, additionalPlannedCapacity=0}

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Selected Region: us-west-2

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Found dynamic agent task definition: arn:aws:ecs:us-west-2:<aws_account>:task-definition/on-prem-agent-cluster-onpremtest68-dz678:1

Jun 17, 2023 9:48:22 PM INFO com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher runECSTask

[on-prem-agent-cluster-onpremtest68-htgt1]: Starting agent with task definition arn:aws:ecs:us-west-2:<aws_account>:task-definition/on-prem-agent-cluster-onpremtest68-dz678:1}

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Selected Region: us-west-2

Jun 17, 2023 9:48:22 PM FINE com.cloudbees.jenkins.plugins.amazonecs.ECSCloud

Found container definition with 1 container(s). Assuming first container is the Jenkins agent: on-prem-agent-cluster-onpremtest68-dz678

Jun 17, 2023 9:48:22 PM WARNING com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher launch

[on-prem-agent-cluster-onpremtest68-htgt1]: Error in provisioning; agent=com.cloudbees.jenkins.plugins.amazonecs.ECSSlave[on-prem-agent-cluster-onpremtest68-htgt1]
com.amazonaws.services.ecs.model.InvalidParameterException: Override argument cannot be null. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: cc490add-edd4-49fa-ae71-5ee09bc50121; Proxy: null)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
    at com.amazonaws.services.ecs.AmazonECSClient.doInvoke(AmazonECSClient.java:5067)
    at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:5034)
    at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:5023)
    at com.amazonaws.services.ecs.AmazonECSClient.executeRunTask(AmazonECSClient.java:3546)
    at com.amazonaws.services.ecs.AmazonECSClient.runTask(AmazonECSClient.java:3515)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSService.runEcsTask(ECSService.java:510)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.runECSTask(ECSLauncher.java:213)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launchECSTask(ECSLauncher.java:149)
    at com.cloudbees.jenkins.plugins.amazonecs.ECSLauncher.launch(ECSLauncher.java:110)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Anything else?

Experiencing strange behavior in the UI when opening the Cloud Configurations, All clouds show as "Edited". Using JCasC to configure an existing cloud and manually adding the cloud that is not functioning. Inspecting the cloud through the script console shows the correct clusterArn. Manually running the created task definition in the ECS cluster through AWS Console/CLI also functions as expected.

jlujan-invitae commented 1 year ago

Creating Task in cluster null is coming from the use of agent.getClusterArn() on ECSLauncher.java:146. This will not work as the clusterArn is not set on the agent until after task creation at https://github.com/jenkinsci/amazon-ecs-plugin/blob/4fe600569f803adb658245a68627907d9933ce9a/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSLauncher.java#L227

It looks like the ECSSlave is tied to the cloud by constructor so getClusterArn() should simply return cloud.getCluster() in https://github.com/jenkinsci/amazon-ecs-plugin/blob/4fe600569f803adb658245a68627907d9933ce9a/src/main/java/com/cloudbees/jenkins/plugins/amazonecs/ECSSlave.java#L102

jlujan-invitae commented 1 year ago

The root cause was the instance Jenkins URL not being set correctly from JCasC upon restart. Opening the global configuration page and saving, then opening the Cloud Configuration and saving appears to have corrected the issue.

This should still be considered a bug in ecs-plugin as getDockerRunCommand appends a null to the command list when the Jenkins URL is null. This results in the InvalidParameterException: Override argument cannot be null and a lot of wasted time.

jenkinsci / amazon-ecs-plugin