curl to notification webhook silently fails in pipeline test_commands

[cworkschris]

Hi, the aws support guys asked me to get in touch with the copilot developers, so here I am! in my copilot/pipelines/mddevpl/manifest.yml I am trying to do a simple notification - which I never get or see results of in logs. I worked with aws support on escaping in various ways to no avail. I have tested the webhook independently to make sure it works.

test_commands:
  - curl -i -H 'Accept:application/json' -H 'Content-Type:application/json' -X POST --data '{"content":"AWS Pipeline"}' https://discord.com/api/webhooks/someid/some_kindo-fguid

[huanjani]

Hi, @cworkschris! Thanks for getting in touch!

Is your pipeline building successfully now? Can we see what the BuildTestCommands[envName] portion of your pipeline's CloudFormation template looks like? (The stack name should be pipeline-[appName]-[chosenPipelineName].) That may help us get to the bottom of this!

Thanks!

[cworkschris]

Thanks for the look!

Oddly, in my: https://us-west-1.console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks

It seems like one was not generated - the list looks like this (none today, the 14th)

even though, the pipeline ran, as can be seen in the list (bottom of the 3):

[cworkschris]

I still don't know why an active pipeline has no associated template, here's the template for that pipeline when it was last recreated (but still no BuildTestCommands even though the copilot test_commands existed then.:

https://us-west-1.console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/template?stackId=arn%3Aaws%3Acloudformation%3Aus-west-1%3A297000690820%3Astack%2Fpipeline-mageduel-mddevpl%2F85249d90-9ba2-11ed-ae2d-0250e1a96a61

# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: MIT-0
AWSTemplateFormatVersion: '2010-09-09'
Description: CodePipeline for mageduel
Resources:

  SourceConnection:
    Type: AWS::CodeStarConnections::Connection
    Properties:
      ConnectionName: copilot-curri-mageduel_backend
      ProviderType: Bitbucket
  BuildProjectRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codebuild.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess' # for env ls
        - 'arn:aws:iam::aws:policy/AWSCloudFormationReadOnlyAccess' # for service package
      Policies:
        - PolicyName: assume-env-manager
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Resource: 'arn:aws:iam::297000690820:role/mageduel-dev-EnvManagerRole'
              Action:
              - sts:AssumeRole
  BuildProjectPolicy:
    Type: AWS::IAM::Policy
    DependsOn: BuildProjectRole
    Properties:
      PolicyName: !Sub ${AWS::StackName}-CodeBuildPolicy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - codebuild:CreateReportGroup
              - codebuild:CreateReport
              - codebuild:UpdateReport
              - codebuild:BatchPutTestCases
              - codebuild:BatchPutCodeCoverages
            Resource: !Sub arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/pipeline-mageduel-*
          - Effect: Allow
            Action:
              - s3:PutObject
              - s3:GetObject
              - s3:GetObjectVersion
            # TODO: This might not be necessary. We may only need the bucket
            # that is in the same region as the pipeline.
            # Loop through all the artifact buckets created in the stackset
            Resource:
              - !Join ['', ['arn:aws:s3:::', 'stackset-mageduel-infras-pipelinebuiltartifactbuc-1dlovg5xhgf3n']]
              - !Join ['', ['arn:aws:s3:::', 'stackset-mageduel-infras-pipelinebuiltartifactbuc-1dlovg5xhgf3n', '/*']]
          - Effect: Allow
            Action:
              # TODO: scope this down if possible
              - kms:*
            # TODO: This might not be necessary. We may only need the KMS key
            # that is in the same region as the pipeline.
            # Loop through all the KMS keys used to en/decrypt artifacts
            # across (cross-regional) pipeline stages, with each stage
            # backed by a (regional) S3 bucket.
            Resource:
              - arn:aws:kms:us-west-1:297000690820:key/d5fe300b-2407-429c-834b-6d8d3ed597ab
          - Effect: Allow
            Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
            Resource: arn:aws:logs:*:*:*
          - Effect: Allow
            Action:
              - ecr:GetAuthorizationToken
            Resource: '*'
          - Effect: Allow
            Action:
              - ecr:DescribeImageScanFindings
              - ecr:GetLifecyclePolicyPreview
              - ecr:GetDownloadUrlForLayer
              - ecr:BatchGetImage
              - ecr:DescribeImages
              - ecr:ListTagsForResource
              - ecr:BatchCheckLayerAvailability
              - ecr:GetLifecyclePolicy
              - ecr:GetRepositoryPolicy
              - ecr:PutImage
              - ecr:InitiateLayerUpload
              - ecr:UploadLayerPart
              - ecr:CompleteLayerUpload
            Resource: '*'
            Condition: {StringEquals: {'ecr:ResourceTag/copilot-application': mageduel}}  
      Roles:
        - !Ref BuildProjectRole
  BuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub ${AWS::StackName}-BuildProject
      Description: !Sub Build for ${AWS::StackName}
      # ArtifactKey is the KMS key ID or ARN that is used with the artifact bucket
      # created in the same region as this pipeline.
      EncryptionKey: !ImportValue mageduel-ArtifactKey
      ServiceRole: !GetAtt BuildProjectRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Cache:
        Modes:
          - LOCAL_DOCKER_LAYER_CACHE
        Type: LOCAL
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        PrivilegedMode: true
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
        EnvironmentVariables:
          - Name: AWS_ACCOUNT_ID
            Value: !Sub '${AWS::AccountId}'
          - Name: PARTITION
            Value: !Ref AWS::Partition
      Source:
        Type: CODEPIPELINE
        BuildSpec: copilot/pipelines/mddevpl/buildspec.yml
      TimeoutInMinutes: 60
  PipelineRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
  PipelineRolePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: !Sub ${AWS::StackName}-CodepipelinePolicy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - codepipeline:*
              - codecommit:GetBranch
              - codecommit:GetCommit
              - codecommit:UploadArchive
              - codecommit:GetUploadArchiveStatus
              - codecommit:CancelUploadArchive
              - iam:ListRoles
              - cloudformation:Describe*
              - cloudFormation:List*
              - codebuild:BatchGetBuilds
              - codebuild:StartBuild
              - cloudformation:CreateStack
              - cloudformation:DeleteStack
              - cloudformation:DescribeStacks
              - cloudformation:UpdateStack
              - cloudformation:CreateChangeSet
              - cloudformation:DeleteChangeSet
              - cloudformation:DescribeChangeSet
              - cloudformation:ExecuteChangeSet
              - cloudformation:SetStackPolicy
              - cloudformation:ValidateTemplate
              - iam:PassRole
              - s3:ListAllMyBuckets
              - s3:GetBucketLocation
            Resource:
              - "*"
          - Effect: Allow
            Action:
              - codestar-connections:CreateConnection
              - codestar-connections:DeleteConnection
              - codestar-connections:GetConnection
              - codestar-connections:ListConnections
              - codestar-connections:GetIndividualAccessToken
              - codestar-connections:GetInstallationUrl
              - codestar-connections:ListInstallationTargets
              - codestar-connections:StartOAuthHandshake
              - codestar-connections:UpdateConnectionInstallation
              - codestar-connections:UseConnection
              - codestar-connections:RegisterAppCode
              - codestar-connections:StartAppRegistrationHandshake
              - codestar-connections:StartUploadArchiveToS3
              - codestar-connections:GetUploadArchiveToS3Status
              - codestar-connections:PassConnection
              - codestar-connections:PassedToService
            Resource:
              - !Ref SourceConnection
          - Effect: Allow
            Action:
              - kms:Decrypt
              - kms:Encrypt
              - kms:GenerateDataKey
            Resource:
              - arn:aws:kms:us-west-1:297000690820:key/d5fe300b-2407-429c-834b-6d8d3ed597ab
          - Effect: Allow
            Action:
              - s3:PutObject
              - s3:GetBucketPolicy
              - s3:GetObject
              - s3:ListBucket
              - s3:PutObjectAcl
              - s3:GetObjectAcl
            Resource:
              - !Join ['', ['arn:aws:s3:::', 'stackset-mageduel-infras-pipelinebuiltartifactbuc-1dlovg5xhgf3n']]
              - !Join ['', ['arn:aws:s3:::', 'stackset-mageduel-infras-pipelinebuiltartifactbuc-1dlovg5xhgf3n', '/*']]
          - Effect: Allow
            Action:
              - sts:AssumeRole
            Resource:
              - arn:aws:iam::297000690820:role/mageduel-dev-EnvManagerRole
      Roles:
        - !Ref PipelineRole
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    DependsOn:
      - PipelineRole
      - PipelineRolePolicy
    Properties:
      ArtifactStores:
        - Region: us-west-1
          ArtifactStore:
            Type: S3
            Location: stackset-mageduel-infras-pipelinebuiltartifactbuc-1dlovg5xhgf3n
            EncryptionKey:
              Id: arn:aws:kms:us-west-1:297000690820:key/d5fe300b-2407-429c-834b-6d8d3ed597ab
              Type: KMS
      RoleArn: !GetAtt PipelineRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: SourceCodeFor-mageduel
              ActionTypeId:
                Category: Source
                Owner: AWS
                Version: 1
                Provider: CodeStarSourceConnection
              Configuration:
                ConnectionArn:
                  !Ref SourceConnection
                FullRepositoryId: curriculaworks/mageduel_backend
                BranchName: dev
              OutputArtifacts:
                - Name: SCCheckoutArtifact
              RunOrder: 1
        - Name: Build
          Actions:
          - Name: Build
            ActionTypeId:
              Category: Build
              Owner: AWS
              Version: 1
              Provider: CodeBuild
            Configuration:
              ProjectName: !Ref BuildProject
            RunOrder: 1
            InputArtifacts:
              - Name: SCCheckoutArtifact
            OutputArtifacts:
              - Name: BuildOutput
        - Name: DeployTo-dev
          Actions:
            - Name: CreateOrUpdate-api-dev
              Region: us-west-1
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: CloudFormation
              Configuration:
                ActionMode: CREATE_UPDATE
                StackName: mageduel-dev-api
                Capabilities: CAPABILITY_IAM,CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND
                TemplatePath: BuildOutput::infrastructure/api-dev.stack.yml
                TemplateConfiguration: BuildOutput::infrastructure/api-dev.params.json
                RoleArn: arn:aws:iam::297000690820:role/mageduel-dev-CFNExecutionRole
              InputArtifacts:
                - Name: BuildOutput
              RunOrder: 1
              RoleArn: arn:aws:iam::297000690820:role/mageduel-dev-EnvManagerRole  
Outputs:
  PipelineConnectionARN:
    Description: "ARN of CodeStar Connections connection"
    Value:
      !Ref SourceConnection

[huanjani]

Hunh! How bizarre. Are you looking in the same region in the CloudFormation and CodePipeline consoles?

[cworkschris]

Ah, ok, I expected it to be at the top of the list, but it is sorted by Creation datetime, so the template listed above was the correct one I think (for pipeline-mageduel-mddevpl)

There is no BuildTestCommands... within!

[huanjani]

@cworkschris-- I just went back and looked at the files I got from the SA last week-- I think your test_commands field in your pipeline manifest is indented with an extra space. It should line up with deployments/name. 🤞🏼 that that does it!

[cworkschris]

Hi Janice, thanks for looking!

I'm a little unsure what you mean - from https://aws.github.io/copilot-cli/docs/concepts/pipelines/ The indentation looks like:

And mine is set up (I think the same type of indentation - at levels 2 and 3):

Do you still think I should back it up so it is even with the previous "-" line?

[Lou1415926]

@cworkschris Hello 👋🏼 Thanks for the screenshot!

From your screenshot, the indentation looks fine!

Can you run copilot pipeline deploy, and show us the template of pipeline-mageduel-mddevpl after you run the command? We are still mostly interested in the keyword BuildTestCommands!

[cworkschris]

Thank you Lou! With the extra copilot pipeline deploy it seems to be working. Sometime in my searches I remember seeing that the environment variables available in copilot/{servicename}/manifest.yml aren't available in the pipeline, but that it was being considered (like ${COPILOT_APPLICATION_NAME} and ${COPILOT_ENVIRONMENT_NAME})

Did that ever happen? (otherwise it is hard to have any context for my notifications)

[Lou1415926]

@cworkschris Would #2428 or #4372 happen to be the threads that you looked at? If so, unfortunately we haven't got to the feature requested in these threads yet : (

Do you need to access environment variables in test_commands or in buildspec.yml? Would the ${COPILOT_APPLICATION_NAME} and ${COPILOT_ENVIRONMENT_NAME} be all you need? There might be a workaround that I can help you with depending on what your goal is!

[cworkschris]

In the test_commands - I'm avoiding manual editing of buildspec.yml if possible.

Those two would be useful - more info would be even better (e.g. git commit info,) but I can live with those two.

Thanks!

[Lou1415926]

In the case of test_commands, unfortunately you might need to hardcode it for now, until we have better support for test_commands with environment variables :(

# In pipelines/mddevpl/manifest.yml
stages:
  - name: dev
    test_commands:
       - echo "mddevpl" # Instead of `echo ${COPILOT_APPLICATION_NAME}`
       - echo "dev" # Instead of `echo ${COPILOT_ENVIRONMENT_NAME}`
  - name: test
    test_commands:
       - echo "mddevpl" # Instead of `echo ${COPILOT_APPLICATION_NAME}`
       - echo "test" # Instead of `echo ${COPILOT_ENVIRONMENT_NAME}`  

Hopefully this is workable for you as a temporary solution :(

Please considering a 👍🏼 to the following issues if you'd like! I think together, they should complete the full features that you need:

• 4372 so that you can inject environment variables into the BuildTestCommands build project

• 4531 so that you can use ${COPILOT_APPLICATION_NAME} in pipeline manifest

[cworkschris]

Ok, thanks! I also +1 https://github.com/aws/copilot-cli/issues/2428