[Bug]: "Cannot unmarshal !!map into string" in pipeline build stage

Description:

I have a working application with 3 microservices, and wanted to set up a CI/CD pipeline to update the user_code service whenever a new change is pushed to our GitHub repo's main branch.

The pipeline passes the "Source" stage, and seems to register new commits very quickly, but fails at the "Build" stage. The logs show that the build is erroring when trying to package the service with an error of "unmarshal workload: unmarshal manifest for Backend Service: yaml"

Details:

Running copilot version: v1.33.3, linux arm64, us-east-1

Pipeline manifest.yml:

```yaml # The manifest for the "etl-orchestration-main-cicd" pipeline. # This YAML file defines your pipeline: the source repository it tracks and the order of the environments to deploy to. # For more info: https://aws.github.io/copilot-cli/docs/manifest/pipeline/ # The name of the pipeline. name: etl-orchestration-main-cicd # The version of the schema used in this template. version: 1 # This section defines your source, changes to which trigger your pipeline. source: # The name of the provider that is used to store the source artifacts. # (i.e. GitHub, Bitbucket, CodeCommit) provider: GitHub # Additional properties that further specify the location of the artifacts. properties: branch: main repository: build: image: aws/codebuild/amazonlinux2-aarch64-standard:2.0 # This section defines the order of the environments your pipeline will deploy to. stages: # The name of the environment - name: prod deployments: user-code: ```

Pipeline buildspec.yml:

```yaml # Buildspec runs in the build stage of your pipeline. version: 0.2 phases: install: commands: - echo "cd into $CODEBUILD_SRC_DIR" - cd $CODEBUILD_SRC_DIR # Download the copilot linux binary - use arm64 to ensure compatability with other services - wget https://ecs-cli-v2-release.s3.amazonaws.com/copilot-linux-arm64-v1.14.0 - mv ./copilot-linux-arm64-v1.14.0 ./copilot-linux - chmod +x ./copilot-linux build: commands: - echo "Run your tests" # - make test post_build: commands: - ls -l - export COLOR="false" - export CI="true" - pipeline=$(cat $CODEBUILD_SRC_DIR/copilot/pipelines/etl-orchestration-main-cicd/manifest.yml | ruby -ryaml -rjson -e 'puts JSON.pretty_generate(YAML.load(ARGF))') - pl_envs=$(echo $pipeline | jq -r '.stages[].name') # Find all the local services in the workspace. - svc_ls_result=$(./copilot-linux svc ls --local --json) - svc_list=$(echo $svc_ls_result | jq '.services') - > if [ ! "$svc_list" = null ]; then svcs=$(echo $svc_ls_result | jq -r '.services[].name'); fi # Find all the local jobs in the workspace. - job_ls_result=$(./copilot-linux job ls --local --json) - job_list=$(echo $job_ls_result | jq '.jobs') - > if [ ! "$job_list" = null ]; then jobs=$(echo $job_ls_result | jq -r '.jobs[].name'); fi # Raise error if no services or jobs are found. - > if [ "$svc_list" = null ] && [ "$job_list" = null ]; then echo "No services or jobs found for the pipeline to deploy. Please create at least one service or job and push the manifest to the remote." 1>&2; exit 1; fi # Generate the cloudformation templates. # The tag is the build ID but we replaced the colon ':' with a dash '-'. # We truncate the tag (from the front) to 128 characters, the limit for Docker tags # (https://docs.docker.com/engine/reference/commandline/tag/) # Check if the `svc package` commanded exited with a non-zero status. If so, echo error msg and exit. - > for env in $pl_envs; do tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev) for svc in $svcs; do ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag; if [ $? -ne 0 ]; then echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2; exit 1; fi done; for job in $jobs; do ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag; if [ $? -ne 0 ]; then echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2; exit 1; fi done; done; - ls -lah ./infrastructure artifacts: files: - "infrastructure/*" ```

user-code manifest.yml:

```yaml # The manifest for the "user-code" service. # Read the full specification for the "Backend Service" type at: # https://aws.github.io/copilot-cli/docs/manifest/backend-service/ # Your service name will be used in naming your resources like log groups, ECS services, etc. name: user-code type: Backend Service # Your service is reachable at "http://user-code.${COPILOT_SERVICE_DISCOVERY_ENDPOINT}:4000" but is not public. # Configuration for your containers and service. image: # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/backend-service/#image-build build: Dockerfile_user_code target: user-code # Port exposed through your container to route traffic to it. port: 4000 healthcheck: command: ["CMD-SHELL", "dagster api grpc-health-check -h 0.0.0.0 -p 4000"] interval: 20s start_period: 20s timeout: 16s cpu: 256 # Number of CPU units for the task. memory: 512 # Amount of memory in MiB used by the task. platform: linux/arm64 # See https://aws.github.io/copilot-cli/docs/manifest/backend-service/#platform count: 1 # Number of tasks that should be running in your service. exec: true # Enable running commands in your container. entry_point: "dagster api grpc -h 0.0.0.0 -p 4000 -m replo_dagster" variables: DAGSTER_CURRENT_IMAGE: "" secrets: DAGSTER_POSTGRES_DB: from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBName DAGSTER_POSTGRES_USER: from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBUser DAGSTER_POSTGRES_HOSTNAME: from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBHost DAGSTER_POSTGRES_PASSWORD: from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBPassword DAGSTER_POSTGRES_PORT: from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBPort # Retrieve Snowflake DB credentials for dbt orchestration DBT_ENV_SECRET_SNOWFLAKE_PROD_USER: "''" DBT_ENV_SECRET_SNOWFLAKE_PROD_PASSWORD: "''" DBT_SNOWFLAKE_PROD_ROLE: "''" network: connect: true # enable Service Connect for intra-environment traffic between services vpc: security_groups: - from_cfn: ${COPILOT_APPLICATION_NAME}-${COPILOT_ENVIRONMENT_NAME}-DagsterDBSecurityGroup ```

Error:

The pipeline build logs show it erroring at this stage:

[Container] 2024/05/23 04:14:42.767184 Running command for env in $pl_envs; do
  tag=$(echo ${CODEBUILD_BUILD_ID##*:}-$env | sed 's/:/-/g' | rev | cut -c 1-128 | rev)
  for svc in $svcs; do
  ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag;
  if [ $? -ne 0 ]; then
    echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
    exit 1;
  fi
  done;
  for job in $jobs; do
  ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag;
  if [ $? -ne 0 ]; then
    echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
    exit 1;
  fi
  done;
done;

✘ unmarshal workload: unmarshal manifest for Backend Service: yaml: unmarshal errors:
  line 23: cannot unmarshal !!map into string
  line 25: cannot unmarshal !!map into string
  line 27: cannot unmarshal !!map into string
  line 29: cannot unmarshal !!map into string
  line 31: cannot unmarshal !!map into string
  line 40: cannot unmarshal !!map into string
Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error.

Debugging:

Based on the error and the number of lines flagged, I assumed it had to do with the lines in my user-code manifest file with "from_cfn". However, I tried referencing the secrets directly from Secrets Manager instead in the backend service and pushed my changes, which would have shortened the number of lines in the file. However the error is the exact same, and I can't seem to figure out what lines these are actually in reference to (since they don't appear to match up to file).

It's also odd that the .yml file seems to be formatted correctly since my actual service deployed fine and is working correctly, but errors out when executing here via the CodeBuild pipeline.

Would greatly appreciate if someone could help me figure out how to resolve this!

aws / copilot-cli