Deploy pipeline succeeds, but it doesn't pick up code changes.

[miguelrios]

I followed the instructions to create a new pipeline for an application with two services. I am able to copilot deploy them separately, but when running the pipeline I created and deployed (using copilot pipeline deploy), I can see in CodePipeline that the deploy succeeds, but the code changes are not reflected in the deployed service.

To test, I made a simple change to an API. I changed:

@app.route("/")
async def homepage(request):
    return JSONResponse({'hello': 'world'})

to

@app.route("/")
async def homepage(request):
    return JSONResponse({'hello': 'hello'})

Then, I pushed the changes to main (using Github). I checked CodePipeline and confirmed all three steps succeeded. the When accessing my endpoint, I still see {'hello': 'world'}.

When running copilot deploy and I deploy the service manually, I see {'hello': 'hello'}.

Here's the relevant files:

copilot/pipelines/buildspec.yml

# Buildspec runs in the build stage of your environment pipeline to generate the environment CloudFormation stack config.
version: 0.2
phases:
  install:
    runtime-versions:
      ruby: 3.1
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      # Download the copilot linux binary. You can update this version.
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.27.0
      - mv ./copilot-linux-v1.27.0 ./copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:
      - echo "Run your tests"
      # - make test
  post_build:
    commands:
      - ls -l
      - export COLOR="false"
      - export CI="true"
      - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/parcha-pipeline/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
      - stages=$(echo $pipeline | jq -r '.stages[].name')
      # Generate the cloudformation templates.
      - >
        for env in $stages; do
          ./copilot-linux env package -n $env --output-dir './infrastructure' --upload-assets --force;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
        done;
      - ls -lah ./infrastructure
artifacts:
  files:
    - "infrastructure/*"

copilot/pipelines/manifest.yml:

# The manifest for the "parcha-pipeline" pipeline.
# This YAML file defines your pipeline: the source repository it tracks and the order of the environments to deploy to.
# For more info: https://aws.github.io/copilot-cli/docs/manifest/pipeline/

# The name of the pipeline.
name: parcha-pipeline

# The version of the schema used in this template.
version: 1

# This section defines your source, changes to which trigger your pipeline.
source:
  # The name of the provider that is used to store the source artifacts.
  # (i.e. GitHub, Bitbucket, CodeCommit)
  provider: GitHub
  # Additional properties that further specify the location of the artifacts.
  properties:
    branch: main
    repository: https://github.com/miguelrios/parcha
    # Optional: specify the name of an existing CodeStar Connections connection.
    # connection_name: a-connection

# This section defines the order of the environments your pipeline will deploy to.
stages:
  - # The name of the environment.
    name: test
    deployments:
      deploy-env:
        template_path: infrastructure/test.env.yml
        template_config: infrastructure/test.env.params.json
        stack_name: parcha-app-test
    # Optional: flag for manual approval action before deployment.
    # requires_approval: true
    # Optional: use test commands to validate this stage of your build.
    # test_commands: [echo 'running tests', make test]

`
And the service manifest.yml:
`
# The manifest for the "parcha-api" service.
# Read the full specification for the "Load Balanced Web Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: parcha-api
type: Load Balanced Web Service

# Distribute traffic to your service.
http:
  # Requests to this path will be forwarded to your service.
  # To match all requests you can use the "/" path.
  path: '/'
  alias: 'api2.parcha.ai'
  # You can specify a custom health check path. The default is "/".
  # healthcheck: '/'

# Configuration for your containers and service.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
  build: Dockerfile.api
  # Port exposed through your container to route traffic to it.
  port: 80

cpu: 1024       # Number of CPU units for the task.
memory: 4096    # Amount of memory in MiB used by the task.
platform: linux/amd64  # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1       # Number of tasks that should be running in your service.
exec: true     # Enable running commands in your container.
network:
  connect: true # Enable Service Connect for intra-environment traffic between services.

# storage:
  # readonly_fs: true       # Limit to read-only access to mounted root filesystems.

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info
env_file: .env

#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.
#    deployment:            # The deployment strategy for the "test" environment.
#       rolling: 'recreate' # Stops existing tasks before new ones are started for faster deployments.

Happy to provide any other relevant info.

[iamhopaul123]

Hello @miguelrios. From the pipeline manifest, it seems like the pipeline is for environments: automating the deployment of Copilot environments by detecting if there's any changes for the env manifests.

From your description it seems like you wanted to deploy a pipeline to automate the service deployment. If that is the case, I would suggest tearing down the existing pipeline and create a new one by following the wizard after running copilot pipeline init.

[miguelrios]

Thanks, @iamhopaul123! I deleted the pipeline and generated one selecting workloads this time. The pipeline is failing in the post build step. Deploying the services manually still work:

✘ upload resources required for deployment for parcha-api: read env file .env: open /codebuild/output/src635344703/src/.env: no such file or directory
--
954 | Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error.
955 |  
956 | [Container] 2023/04/06 16:35:28 Command did not exit successfully for env in $pl_envs; do
957 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
958 | for svc in $svcs; do
959 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
960 | if [ $? -ne 0 ]; then
961 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
962 | exit 1;
963 | fi
964 | done;
965 | for job in $jobs; do
966 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
967 | if [ $? -ne 0 ]; then
968 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
969 | exit 1;
970 | fi
971 | done;
972 | done;
973 | exit status 1
974 | [Container] 2023/04/06 16:35:28 Phase complete: POST_BUILD State: FAILED
975 | [Container] 2023/04/06 16:35:28 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs; do
976 | tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env \| sed 's/:/-/g' \| rev \| cut -c 1-128 \| rev)
977 | for svc in $svcs; do
978 | ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
979 | if [ $? -ne 0 ]; then
980 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
981 | exit 1;
982 | fi
983 | done;
984 | for job in $jobs; do
985 | ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
986 | if [ $? -ne 0 ]; then
987 | echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
988 | exit 1;
989 | fi
990 | done;
991 | done;
992 | . Reason: exit status 1
993 | [Container] 2023/04/06 16:35:28 Expanding base directory path: .
994 | [Container] 2023/04/06 16:35:28 Assembling file list
995 | [Container] 2023/04/06 16:35:28 Expanding .
996 | [Container] 2023/04/06 16:35:28 Expanding file paths for base directory .
997 | [Container] 2023/04/06 16:35:28 Assembling file list
998 | [Container] 2023/04/06 16:35:28 Expanding infrastructure/*
999 | [Container] 2023/04/06 16:35:28 Found 2 file(s)
1000 | [Container] 2023/04/06 16:35:28 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
1001 | [Container] 2023/04/06 16:35:28 Phase context status code:  Message:

The manifest file:

# The manifest for the "parcha-cicd" pipeline.
# This YAML file defines your pipeline: the source repository it tracks and the order of the environments to deploy to.
# For more info: https://aws.github.io/copilot-cli/docs/manifest/pipeline/

# The name of the pipeline.
name: parcha-cicd

# The version of the schema used in this template.
version: 1

# This section defines your source, changes to which trigger your pipeline.
source:
  # The name of the provider that is used to store the source artifacts.
  # (i.e. GitHub, Bitbucket, CodeCommit)
  provider: GitHub
  # Additional properties that further specify the location of the artifacts.
  properties:
    branch: main
    repository: https://github.com/miguelrios/parcha
    # Optional: specify the name of an existing CodeStar Connections connection.
    # connection_name: a-connection

# This section defines the order of the environments your pipeline will deploy to.
stages:
  - # The name of the environment.
    name: test
    # Optional: flag for manual approval action before deployment.
    # requires_approval: true
    # Optional: use test commands to validate this stage of your build.
    # test_commands: [echo 'running tests', make test]

And the buildspec file:

# Buildspec runs in the build stage of your pipeline.
version: 0.2
phases:
  install:
    runtime-versions:
      docker: 20
      ruby: 3.1
      nodejs: 16
    commands:
      - echo "cd into $CODEBUILD_SRC_DIR"
      - cd $CODEBUILD_SRC_DIR
      # Download the copilot linux binary.
      - wget -q https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-v1.27.0
      - mv ./copilot-linux-v1.27.0 ./copilot-linux
      - chmod +x ./copilot-linux
  build:
    commands:
      - echo "Run your tests"
      # - make test
  post_build:
    commands:
      - ls -l
      - export COLOR="false"
      - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/parcha-cicd/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))')
      - pl_envs=$(echo $pipeline | jq -r '.stages[].name')
      # Find all the local services in the workspace.
      - svc_ls_result=$(./copilot-linux svc ls --local --json)
      - svc_list=$(echo $svc_ls_result | jq '.services')
      - >
        if [ ! "$svc_list" = null ]; then
          svcs=$(echo $svc_ls_result | jq -r '.services[].name');
        fi
      # Find all the local jobs in the workspace.
      - job_ls_result=$(./copilot-linux job ls --local --json)
      - job_list=$(echo $job_ls_result | jq '.jobs')
      - >
        if [ ! "$job_list" = null ]; then
          jobs=$(echo $job_ls_result | jq -r '.jobs[].name');
        fi
      # Raise error if no services or jobs are found.
      - >
        if [ "$svc_list" = null ] && [ "$job_list" = null ]; then
          echo "No services or jobs found for the pipeline to deploy. Please create at least one service or job and push the manifest to the remote." 1>&2;
          exit 1;
        fi
      # Generate the cloudformation templates.
      # The tag is the build ID but we replaced the colon ':' with a dash '-'.
      # We truncate the tag (from the front) to 128 characters, the limit for Docker tags
      # (https://docs.docker.com/engine/reference/commandline/tag/)
      # Check if the `svc package` commanded exited with a non-zero status. If so, echo error msg and exit.
      - >
        for env in $pl_envs; do
          tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev)
          for svc in $svcs; do
          ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
          done;
          for job in $jobs; do
          ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
          if [ $? -ne 0 ]; then
            echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
            exit 1;
          fi
          done;
        done;
      - ls -lah ./infrastructure
artifacts:
  files:
    - "infrastructure/*"

[miguelrios]

The workload manifest has a pointer to the env file already (env_file: .env):

# The manifest for the "parcha-api" service.
# Read the full specification for the "Load Balanced Web Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: parcha-api
type: Load Balanced Web Service

# Distribute traffic to your service.
http:
  # Requests to this path will be forwarded to your service.
  # To match all requests you can use the "/" path.
  path: '/'
  alias: 'api2.parcha.ai'
  # You can specify a custom health check path. The default is "/".
  # healthcheck: '/'

# Configuration for your containers and service.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
  build: Dockerfile.api
  # Port exposed through your container to route traffic to it.
  port: 80

cpu: 1024       # Number of CPU units for the task.
memory: 4096    # Amount of memory in MiB used by the task.
platform: linux/amd64  # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1       # Number of tasks that should be running in your service.
exec: true     # Enable running commands in your container.
network:
  connect: true # Enable Service Connect for intra-environment traffic between services.

# storage:
  # readonly_fs: true       # Limit to read-only access to mounted root filesystems.

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info
env_file: .env

#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.
#    deployment:            # The deployment strategy for the "test" environment.
#       rolling: 'recreate' # Stops existing tasks before new ones are started for faster deployments.

[miguelrios]

I'm assuming this is because my .env file is not in the repo (by design). I will try to push a dummy one and see if it works. If it does, I guess I will need to use secrets manager or some other way of storing my api keys so it works with copilot. I'll update here.

[iamhopaul123]

yeah for env_file: .env, .env has to be a valid path so that copilot can help you upload from your local machine to our cache s3 bucket and referred to that in the task definition.

[miguelrios]

It is a valid path locally, but it is not uploaded to the github repo (by design). I added an empty .env file and CodeDeploy works. But now I don't have an env file :).

[miguelrios]

Ok that was the issue. I will figure out a better way of handling keys. Thank you for your help, @iamhopaul123!