COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs;

[BaileyNepe]

COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs;

I have come across this issue a couple times now. The first one succeeded after a retry while the second one took a couple retries for it to work. I did not make any changes between retries.

Build logs:

Step 1/19 : FROM node:18.12-alpine3.16 as build
--
134 | toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
135 | upload resources required for deployment for api: build and push image: build Dockerfile at /codebuild/output/src475258656/src/Dockerfile.server: building image: exit status 1
136 | Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error.
137 |  
138 | [Container] 2023/05/15 03:26:31 Command did not exit successfully for env in $pl_envs; do
139 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
140 | for svc in $svcs; do
141 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
142 | if [ $? -ne 0 ]; then
143 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
144 | exit 1;
145 | fi
146 | done;
147 | for job in $jobs; do
148 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
149 | if [ $? -ne 0 ]; then
150 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
151 | exit 1;
152 | fi
153 | done;
154 | done;
155 | exit status 1
156 | [Container] 2023/05/15 03:26:31 Phase complete: POST_BUILD State: FAILED
157 | [Container] 2023/05/15 03:26:31 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs; do
158 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
159 | for svc in $svcs; do
160 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
161 | if [ $? -ne 0 ]; then
162 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
163 | exit 1;
164 | fi
165 | done;
166 | for job in $jobs; do
167 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
168 | if [ $? -ne 0 ]; then
169 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
170 | exit 1;
171 | fi
172 | done;
173 | done;
174 | . Reason: exit status 1
175 | [Container] 2023/05/15 03:26:31 Expanding base directory path: .
176 | [Container] 2023/05/15 03:26:31 Assembling file list
177 | [Container] 2023/05/15 03:26:31 Expanding .
178 | [Container] 2023/05/15 03:26:31 Expanding file paths for base directory .
179 | [Container] 2023/05/15 03:26:31 Assembling file list
180 | [Container] 2023/05/15 03:26:31 Expanding infrastructure/*
181 | [Container] 2023/05/15 03:26:31 Found 2 file(s)
182 | [Container] 2023/05/15 03:26:31 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
183 | [Container] 2023/05/15 03:26:31 Phase context status code:  Message:

pipeline buildspec:

# Buildspec runs in the build stage of your pipeline.
version: 0.2
phases:
  install:
    runtime-versions:
      docker: 19
      ruby: 2.6
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      # Download the copilot linux binary.
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.26.0
      - mv ./copilot-linux-v1.26.0 ./copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:
      - echo "Run your tests"
      # - make test
  post_build:
    commands:
      - ls -l
      - export COLOR="false"
      - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/production/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
      - pl_envs=$(echo $pipeline | jq -r '.stages[].name')
      # Find all the local services in the workspace.
      - svc_ls_result=$(./copilot-linux svc ls --local --json)
      - svc_list=$(echo $svc_ls_result | jq '.services')
      - >
        if [ ! "$svc_list" = null ]; then
          svcs=$(echo $svc_ls_result | jq -r '.services[].name');
        fi
      # Find all the local jobs in the workspace.
      - job_ls_result=$(./copilot-linux job ls --local --json)
      - job_list=$(echo $job_ls_result | jq '.jobs')
      - >
        if [ ! "$job_list" = null ]; then
          jobs=$(echo $job_ls_result | jq -r '.jobs[].name');
        fi
      # Raise error if no services or jobs are found.
      - >
        if [ "$svc_list" = null ] && [ "$job_list" = null ]; then
          echo "No services or jobs found for the pipeline to deploy. Please create at least one service or job and push the manifest to the remote." 1>&2;
          exit 1;
        fi
      # Generate the cloudformation templates.
      # The tag is the build ID but we replaced the colon ':' with a dash '-'.
      # We truncate the tag (from the front) to 128 characters, the limit for Docker tags
      # (https://docs.docker.com/engine/reference/commandline/tag/)
      # Check if the `svc package` commanded exited with a non-zero status. If so, echo error msg and exit.
      - >
        for env in $pl_envs; do
          tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev)
          for svc in $svcs; do
          ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
          done;
          for job in $jobs; do
          ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
          done;
        done;
      - ls -lah ./infrastructure
artifacts:
  files:
    - "infrastructure/*"

svc manifest:

name: api
type: Load Balanced Web Service

http:
  path: '/'
  healthcheck: '/status'

image:
  build: Dockerfile.server
  port: 4100

cpu: 512
memory: 1024
platform: linux/x86_64
count:
  range: 1-4
  cooldown:
    in: 60s
    out: 30s
  cpu_percentage: 70
  memory_percentage: 80
exec: true

network:
  connect: false

storage:
  ephemeral: 20
  volumes:
    logs-volume:
      path: /app/server/logs

environments:
  dev:
    count:
      range: 1-4

  prod:
    cpu: 1024
    memory: 4096
    count:
      range: 2-8
    sidecars:
      datadog:
        image: public.ecr.aws/datadog/agent:latest

[huanjani]

Hello, @BaileyNepe.

Sorry for the delay. It looks like you're running into Docker's pull rate limit. Please see this comment on a previous GH issue for a workaround. Let us know how that goes!

Thanks.

[huanjani]

Closing this issue; if you have other questions or run into another issue, feel free to reopen it!

[BaileyNepe]

Thanks @huanjani

[ssyberg]

I get this issue all the time, I can deploy fine locally, but the copilot generated pipeline basically never works (it's worked a few times but I don't know why). It's alway the POST_BUILD stage. It kinda looks like there's some unmarshalling error in there but copilot svc package runs without issue on all my services AND again, it works if I do a local copilot svc deploy so unclear to me why the pipeline would fail...

[Container] 2023/07/20 13:44:42 Waiting for agent ping
--
2 | [Container] 2023/07/20 13:44:43 Waiting for DOWNLOAD_SOURCE
3 | [Container] 2023/07/20 13:44:45 Phase is DOWNLOAD_SOURCE
4 | [Container] 2023/07/20 13:44:45 CODEBUILD_SRC_DIR=/codebuild/output/src342920781/src
5 | [Container] 2023/07/20 13:44:45 YAML location is /codebuild/output/src342920781/src/copilot/pipelines/****-staging/buildspec.yml
6 | [Container] 2023/07/20 13:44:45 Setting HTTP client timeout to higher timeout for S3 source
7 | [Container] 2023/07/20 13:44:45 Processing environment variables
8 | [Container] 2023/07/20 13:44:45 Selecting 'nodejs' runtime version '16' based on manual selections...
9 | [Container] 2023/07/20 13:44:45 Selecting 'ruby' runtime version '3.1' based on manual selections...
10 | [Container] 2023/07/20 13:44:47 Running command echo "Installing Ruby version 3.1 ..."
11 | Installing Ruby version 3.1 ...
12 |  
13 | [Container] 2023/07/20 13:44:47 Running command rbenv global $RUBY_31_VERSION
14 |  
15 | [Container] 2023/07/20 13:44:47 Moving to directory /codebuild/output/src342920781/src
16 | [Container] 2023/07/20 13:44:47 Configuring ssm agent with target id: codebuild:2f9a7fc6-1871-4167-a1b8-490d5b97ee65
17 | [Container] 2023/07/20 13:44:47 Successfully updated ssm agent configuration
18 | [Container] 2023/07/20 13:44:47 Registering with agent
19 | [Container] 2023/07/20 13:44:47 Phases found in YAML: 3
20 | [Container] 2023/07/20 13:44:47  BUILD: 1 commands
21 | [Container] 2023/07/20 13:44:47  POST_BUILD: 13 commands
22 | [Container] 2023/07/20 13:44:47  INSTALL: 5 commands
23 | [Container] 2023/07/20 13:44:47 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
24 | [Container] 2023/07/20 13:44:47 Phase context status code:  Message:
25 | [Container] 2023/07/20 13:44:47 Entering phase INSTALL
26 | [Container] 2023/07/20 13:44:47 Running command echo "cd into $CODEBUILD_SRC_DIR"
27 | cd into /codebuild/output/src342920781/src
28 |  
29 | [Container] 2023/07/20 13:44:47 Running command cd $CODEBUILD_SRC_DIR
30 |  
31 | [Container] 2023/07/20 13:44:47 Running command wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.28.0
32 |  
33 | [Container] 2023/07/20 13:44:50 Running command mv ./copilot-linux-v1.28.0 ./copilot-linux
34 |  
35 | [Container] 2023/07/20 13:44:50 Running command chmod +x ./copilot-linux
36 |  
37 | [Container] 2023/07/20 13:44:50 Phase complete: INSTALL State: SUCCEEDED
38 | [Container] 2023/07/20 13:44:50 Phase context status code:  Message:
39 | [Container] 2023/07/20 13:44:50 Entering phase PRE_BUILD
40 | [Container] 2023/07/20 13:44:50 Phase complete: PRE_BUILD State: SUCCEEDED
41 | [Container] 2023/07/20 13:44:50 Phase context status code:  Message:
42 | [Container] 2023/07/20 13:44:50 Entering phase BUILD
43 | [Container] 2023/07/20 13:44:50 Running command echo "Run your tests"
44 | Run your tests
45 |  
46 | [Container] 2023/07/20 13:44:50 Phase complete: BUILD State: SUCCEEDED
47 | [Container] 2023/07/20 13:44:50 Phase context status code:  Message:
48 | [Container] 2023/07/20 13:44:50 Entering phase POST_BUILD
49 | [Container] 2023/07/20 13:44:50 Running command ls -l
50 | total 44744
51 | -rw-rw-r--  1 root root       17 Jul 20 13:43 Dockerfile.mongo
52 | -rw-rw-r--  1 root root      129 Jul 20 13:43 Dockerfile.test
53 | -rw-rw-r--  1 root root     1980 Jul 20 13:43 README.md
54 | drwxr-xr-x  3 root root      237 Jul 20 13:44 backend
55 | drwxr-xr-x  6 root root       92 Jul 20 13:44 copilot
56 | -rwxr-xr-x  1 root root 45686784 May 24 20:06 copilot-linux
57 | -rw-rw-r--  1 root root     3411 Jul 20 13:43 docker-compose.yml
58 | -rw-rw-r--  1 root root      141 Jul 20 13:43 favicon.ico
59 | drwxr-xr-x 11 root root     4096 Jul 20 13:44 frontend
60 | -rw-rw-r--  1 root root      415 Jul 20 13:43 package.json
61 | -rwxrwxr-x  1 root root      598 Jul 20 13:43 test.sh
62 | -rw-rw-r--  1 root root    95969 Jul 20 13:43 yarn.lock
63 |  
64 | [Container] 2023/07/20 13:44:50 Running command export COLOR="false"
65 |  
66 | [Container] 2023/07/20 13:44:50 Running command pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/*****-staging/manifest.yml \| ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
67 |  
68 | [Container] 2023/07/20 13:44:53 Running command pl_envs=$(echo $pipeline \| jq -r '.stages[].name')
69 |  
70 | [Container] 2023/07/20 13:44:53 Running command svc_ls_result=$(./copilot-linux svc ls --local --json)
71 |  
72 | [Container] 2023/07/20 13:44:53 Running command svc_list=$(echo $svc_ls_result \| jq '.services')
73 |  
74 | [Container] 2023/07/20 13:44:53 Running command if [ ! "$svc_list" = null ]; then
75 | svcs=$(echo $svc_ls_result \| jq -r '.services[].name');
76 | fi
77 |  
78 |  
79 | [Container] 2023/07/20 13:44:53 Running command job_ls_result=$(./copilot-linux job ls --local --json)
80 |  
81 | [Container] 2023/07/20 13:44:53 Running command job_list=$(echo $job_ls_result \| jq '.jobs')
82 |  
83 | [Container] 2023/07/20 13:44:53 Running command if [ ! "$job_list" = null ]; then
84 | jobs=$(echo $job_ls_result \| jq -r '.jobs[].name');
85 | fi
86 |  
87 |  
88 | [Container] 2023/07/20 13:44:53 Running command if [ "$svc_list" = null ] && [ "$job_list" = null ]; then
89 | echo "No services or jobs found for the pipeline to deploy. Please create at least one service or job and push the manifest to the remote." 1>&2;
90 | exit 1;
91 | fi
92 |  
93 |  
94 | [Container] 2023/07/20 13:44:53 Running command for env in $pl_envs; do
95 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
96 | for svc in $svcs; do
97 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
98 | if [ $? -ne 0 ]; then
99 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
100 | exit 1;
101 | fi
102 | done;
103 | for job in $jobs; do
104 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
105 | if [ $? -ne 0 ]; then
106 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
107 | exit 1;
108 | fi
109 | done;
110 | done;
111 |  
112 | ✘ unmarshal service backend manifest: unmarshal manifest for Load Balanced Web Service: cannot marshal "http" field into bool or map
113 | Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error.
114 |  
115 | [Container] 2023/07/20 13:44:53 Command did not exit successfully for env in $pl_envs; do
116 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
117 | for svc in $svcs; do
118 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
119 | if [ $? -ne 0 ]; then
120 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
121 | exit 1;
122 | fi
123 | done;
124 | for job in $jobs; do
125 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
126 | if [ $? -ne 0 ]; then
127 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
128 | exit 1;
129 | fi
130 | done;
131 | done;
132 | exit status 1
133 | [Container] 2023/07/20 13:44:53 Phase complete: POST_BUILD State: FAILED
134 | [Container] 2023/07/20 13:44:53 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs; do
135 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
136 | for svc in $svcs; do
137 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
138 | if [ $? -ne 0 ]; then
139 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
140 | exit 1;
141 | fi
142 | done;
143 | for job in $jobs; do
144 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
145 | if [ $? -ne 0 ]; then
146 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
147 | exit 1;
148 | fi
149 | done;
150 | done;
151 | . Reason: exit status 1
152 | [Container] 2023/07/20 13:44:53 Expanding base directory path: .
153 | [Container] 2023/07/20 13:44:53 Assembling file list
154 | [Container] 2023/07/20 13:44:53 Expanding .
155 | [Container] 2023/07/20 13:44:53 Expanding file paths for base directory .
156 | [Container] 2023/07/20 13:44:53 Assembling file list
157 | [Container] 2023/07/20 13:44:53 Expanding infrastructure/*
158 | [Container] 2023/07/20 13:44:53 Found 2 file(s)
159 | [Container] 2023/07/20 13:44:53 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
160 | [Container] 2023/07/20 13:44:53 Phase context status code:  Message:

[ssyberg]

FWIW I just updated my local copilot to 1.29.0 and ran copilot app upgrade but I dont see any local changes as a result and copilot svc package still works. It appears there's some discrepancy between copilot and copilot-linux... if so perhaps this should be a new bug?

[huanjani]

Hi, @ssyberg! Would you mind posting your Load Balanced Web Service manifest here? It looks like there may be an issue with the http field.

[ssyberg]

Sure, see below, but just reiterating copilot svc deploy works just fine, it's only in the pipeline that it fails.

# The manifest for the "backend" service.
# Read the full specification for the "Load Balanced Web Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: backend
type: Load Balanced Web Service

# Distribute traffic to your service.
http:
    # Requests to this path will be forwarded to your service.
    # To match all requests you can use the "/" path.
    alias: ${COPILOT_ENVIRONMENT_NAME}.example.com
    path: 'cms'
    healthcheck:
    path: '/healthcheck'
    start_period: 60s
    protocol: http

    additional_rules:
        - path: 'static'
          alias: ${COPILOT_ENVIRONMENT_NAME}.example.com
          healthcheck:
              path: '/healthcheck'
              start_period: 60s
              protocol: http

# Configuration for your containers and service.
image:
    # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
    build: backend/Dockerfile.deploy
    # Port exposed through your container to route traffic to it.
    port: 3000

cpu: 2048 # Number of CPU units for the task.
memory: 4096 # Amount of memory in MiB used by the task.
platform: linux/x86_64 # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1 # Number of tasks that should be running in your service.
exec: true # Enable running commands in your container.
network:
    connect: true # Enable Service Connect for intra-environment traffic between services.
    vpc:
        placement: 'private'

# storage:
# readonly_fs: true       # Limit to read-only access to mounted root filesystems.

# Optional fields for more advanced use-cases.
#
variables: # Pass environment variables as key value pairs.
    [REDACTED]

#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.
secrets:
    [REDACTED]

deployment:
    rolling: 'recreate'
# You can override any of the values defined above by environment.
# environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.
#    deployment:            # The deployment strategy for the "test" environment.
#       rolling: 'recreate' # Stops existing tasks before new ones are started for faster deployments.
environments:
    develop:
        image:
            # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
            build: backend/Dockerfile.develop.deploy
        variables:
            PORT: 3000
            NODE_ENV: development

[huanjani]

What's your OS/arch?

[huanjani]

Re: your manifest:

• the subfields of http.healthcheck need to be indented
• start_period is a subfield under image.healthcheck, not http
• we don't have a protocol manifest field

Please see more here: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

Hope that helps and let me know if you have questions!

[ssyberg]

Thanks, I'll try making these changes! Just a little feedback, the lack of any tool to expose these issues leads to a situation where we are all just trying one thing after another, cobbling together suggestions from the internet, until something seems to work. IMO this should be the #1 priority of any further CLI work, I think it would alleviate a ton of support work for your team as well!

[Taronm98]

The issue arises when attempting to loop through $pl_envs while having only one environment created. To address this, consider either creating additional environments or adjusting the YAML file to function seamlessly with a single environment.