Cannot update canary Environment Variables with a role's Federated Principal

[saugion]

Describe the bug

When running the aws synthetics update-canary command, It seems that the Canary is updated, since the timestamp changes, but the Environment variables are not updated. I'm using a role created through the gitlab oidc. The referred role's policy has "synthetics:UpdateCanary", plus all the mentioned in the doc: s3:PutObject, s3:GetBucketLocation, s3:ListAllMyBuckets, cloudwatch:PutMetricData, logs:CreateLogGroup, logs:CreateLogStream. I have also tried with --execution-role-arn my-role, that has not a federated principal but "Service": "lambda.amazonaws.com", but no changes.

Expected Behavior

Running: aws synthetics update-canary --name "monitoring-test" --run-config "EnvironmentVariables={test1=test1}" --region "eu-central-1"

Returning: success message with code 200 and empty body.

Timestamp for last updated is changed.

Canary script successfully updated.

Current Behavior

Running: aws synthetics update-canary --name "monitoring-test" --run-config "EnvironmentVariables={test1=test1}" --region "eu-central-1"

Returning: success message with code 200 and empty body.

Timestamp for last updated is changed.

Env variables not updated.

Reproduction Steps

Create canary and use S3 as data source for the script (I have used the terraform resource to initially create the Canary.). Verify that the Canary is created and runs successfully.

Create a gitlab oidc connection (https://docs.gitlab.com/ee/ci/cloud_services/index.html)

Run the aws synthetics update-canary CLI command to make Canary pull new script: aws synthetics update-canary --name "monitoring-test" --run-config "EnvironmentVariables={test1=test1}" --region "eu-central-1"

Possible Solution

I'm not sure how to fix this issue, but It would be great to have some form sort of an error message if the update command is actually failing.

Additional Information/Context

Using the credentials of my user, the same command works fine. The logs are exactly the same in both cases.

CLI version used 2.11.12

[saugion]

I've also tried with the amazon sdk: i can delete, but i cannot update. When i update, the last modified date changes, but the data does't get updated

SyntheticsClient syntheticsClient = SyntheticsClient.builder()
                .region(Region.of("eu-central-1"))
                .credentialsProvider(
                        AwsCredentialsProviderChain.builder().credentialsProviders(StaticCredentialsProvider.create(AwsSessionCredentials.create(
                        "xxxx",
                                "xxxx",
                                "xxx")))
                                .build())
                .build();

        UpdateCanaryRequest updateCanaryRequest = UpdateCanaryRequest.builder()
                .name("test")
                .failureRetentionPeriodInDays(1)
                .runConfig(CanaryRunConfigInput.builder().environmentVariables(Map.of("test1", "test1")).build())
                .build();

        syntheticsClient.updateCanary(updateCanaryRequest);

        syntheticsClient.deleteCanary(DeleteCanaryRequest.builder().name("test").build());

[RyanFitzSimmonsAK]

Hi @saulgiordani, thanks for reaching out. Could you update your AWS CLI version, and try again? If that doesn't work, could you provide debug logs of this behavior? You can get debug logs by adding --debug to your CLI command, and redacting any sensitive information. Thanks!

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[saugion]

I'm sorry I forgot to answer. The issue was resolved thanks to the aws support.

Basically when you create a canary, a canary role is created automatically if not exists, and it is a service-role. This means the fullaccesspolicy applies out of the box.

If you create a custom role, this won't be a service-role, and you need to define the role in the access policy to make it work, check "Action":[ "iam:PassRole" ], "Resource":[ "arn:aws:iam:::role/service-role/CloudWatchSyntheticsRole" ]

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "synthetics:*"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:CreateBucket",
            "s3:PutEncryptionConfiguration"
         ],
         "Resource":[
            "arn:aws:s3:::cw-syn-results-*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListRoles",
            "s3:ListAllMyBuckets",
            "s3:GetBucketLocation",
            "xray:GetTraceSummaries",
            "xray:BatchGetTraces",
            "apigateway:GET"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObject",
            "s3:ListBucket"
         ],
         "Resource":"arn:aws:s3:::cw-syn-*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersion"
         ],
         "Resource":"arn:aws:s3:::aws-synthetics-library-*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:PassRole"
         ],
         "Resource":[
            "arn:aws:iam::*:role/service-role/CloudWatchSyntheticsRole*"
         ],
         "Condition":{
            "StringEquals":{
               "iam:PassedToService":[
                  "lambda.amazonaws.com",
                  "synthetics.amazonaws.com"
               ]
            }
         }
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:GetRole"
         ],
         **"Resource":[
            "arn:aws:iam::*:role/service-role/CloudWatchSyntheticsRole*"
         ]**
      },
      {
         "Effect":"Allow",
         "Action":[
            "cloudwatch:GetMetricData",
            "cloudwatch:GetMetricStatistics"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "cloudwatch:PutMetricAlarm",
            "cloudwatch:DeleteAlarms"
         ],
         "Resource":[
            "arn:aws:cloudwatch:*:*:alarm:Synthetics-*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "cloudwatch:DescribeAlarms"
         ],
         "Resource":[
            "arn:aws:cloudwatch:*:*:alarm:*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "lambda:CreateFunction",
            "lambda:AddPermission",
            "lambda:PublishVersion",
            "lambda:UpdateFunctionConfiguration",
            "lambda:GetFunctionConfiguration"
         ],
         "Resource":[
            "arn:aws:lambda:*:*:function:cwsyn-*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "lambda:GetLayerVersion",
            "lambda:PublishLayerVersion"
         ],
         "Resource":[
            "arn:aws:lambda:*:*:layer:cwsyn-*",
            "arn:aws:lambda:*:*:layer:Synthetics:*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeVpcs",
            "ec2:DescribeSubnets",
            "ec2:DescribeSecurityGroups"
         ],
         "Resource":[
            "*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "sns:ListTopics"
         ],
         "Resource":[
            "*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "sns:CreateTopic",
            "sns:Subscribe",
            "sns:ListSubscriptionsByTopic"
         ],
         "Resource":[
            "arn:*:sns:*:*:Synthetics-*"
         ]
      }
   ]
}

[saugion]

Said that, i still think the logs should reflect the misconfiguration, or at least not return a 200.

[RyanFitzSimmonsAK]

To clarify before I raise this with the service team, your role lacked the permissions to update the canary, and rather than giving an error, it executed successfully and just didn't update it?

[saugion]

Basically yes. My role has the UpdateCanary permission, but the resource connected to the GetRole policy was not correctly set. The logs reflected a 200 and this should not happen, because you are not updating anything.

[RyanFitzSimmonsAK]

I've reached out to the service team about this behavior.

Ticket # for internal use : P101973576

[RyanFitzSimmonsAK]

This appears to have been fixed, unless I'm misunderstanding the issue.

An error occurred (AccessDeniedException) when calling the UpdateCanary operation: User: [myUser] is not authorized to perform: synthetics:UpdateCanary on resource: [myCanary]

Please let me know if you're still having an issue with this operation.

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.