search

aws-observability / application-signals-demo

Apache License 2.0
24 stars 18 forks source link

EKS demo: create-slo.sh fails, as no services available to cloudwatch #65

Open str3tch opened 6 days ago

str3tch commented 6 days ago

This is using the EKS demo version of the repo.

The setup fails at at the create-slo.sh stage.

The reason it fails is because there are no services available for the list-services command to list:

++ aws application-signals list-services --endpoint https://application-signals.ap-southeast-1.api.aws --region ap-southeast-1 --cli-input-json '{
  "StartTime": 1729037340,
  "EndTime": 1729123740
}' --output json --query '(ServiceSummaries[?KeyAttributes.Name=='\''pet-clinic-frontend-java'\''].KeyAttributes)[0]'
+ SERVICE_KEY_ATTRIBUTES=null
+ echo null
null

Here is the full output of the script.

Admin:~/environment/application-signals-demo/scripts/eks/appsignals/one-step (main) $ ../create-slo.sh unicorn-pet-rentals ap-southeast-1
++ dirname ../create-slo.sh
+ cd ..
+ CLUSTER_NAME=unicorn-pet-rentals
+ REGION=ap-southeast-1
+ SERVICE_NAME=pet-clinic-frontend-java
+ ENDPOINT=https://application-signals.ap-southeast-1.api.aws
+ echo 'Creating Service Level Objectives'
Creating Service Level Objectives
++ date +%s
+ end_time=1729123740
+ end_time_as_int=1729123740
+ start_time_as_int=1729037340
+ LIST_SERVICES_REQUEST=slo/inputRequest/ListServices/listServices.json
++ sed -e 's|"StartTime": .*|"StartTime": 1729037340,|' -e 's|"EndTime": .*|"EndTime": 1729123740|' slo/inputRequest/ListServices/listServices.json
+ LIST_SERVICES_REQUEST_WITH_CORRECT_INPUT='{
  "StartTime": 1729037340,
  "EndTime": 1729123740
}'
++ aws application-signals list-services --endpoint https://application-signals.ap-southeast-1.api.aws --region ap-southeast-1 --cli-input-json '{
  "StartTime": 1729037340,
  "EndTime": 1729123740
}' --output json --query '(ServiceSummaries[?KeyAttributes.Name=='\''pet-clinic-frontend-java'\''].KeyAttributes)[0]'
+ SERVICE_KEY_ATTRIBUTES=null
+ echo null
null
+ '[' null = '' ']'
+ KEY_ATTRIBUTES=null
+ CREATE_SLO_REQUEST_1=slo/inputRequest/CreateServiceLevelObjective/getOwner99Availability.json
++ sed 's|"KeyAttributes": .*|"KeyAttributes": null,|' slo/inputRequest/CreateServiceLevelObjective/getOwner99Availability.json
+ CREATE_SLO_REQUEST_WITH_CORRECT_SERVICE_ARN_1='{
  "Name": "Availability for Searching an Owner",
  "Description": "Availability larger than 99 for Get Owner operation",
  "SliConfig": {
    "SliMetricConfig": {
      "KeyAttributes": null,
      "OperationName": "GET /api/customer/owners",
      "MetricType": "AVAILABILITY",
      "PeriodSeconds": 60
    },
    "MetricThreshold": 99.0,
    "ComparisonOperator": "GreaterThan"
  },
  "Goal": {
    "Interval": {
      "RollingInterval": {
        "Duration": 1,
        "DurationUnit": "DAY"
      }
    },
    "AttainmentGoal": 99.9,
    "WarningThreshold": 60.0
  }
}'
+ err=0
+ for i in {1..5}
++ aws application-signals create-service-level-objective --endpoint https://application-signals.ap-southeast-1.api.aws --cli-input-json '{
  "Name": "Availability for Searching an Owner",
  "Description": "Availability larger than 99 for Get Owner operation",
  "SliConfig": {
    "SliMetricConfig": {
      "KeyAttributes": null,
      "OperationName": "GET /api/customer/owners",
      "MetricType": "AVAILABILITY",
      "PeriodSeconds": 60
    },
    "MetricThreshold": 99.0,
    "ComparisonOperator": "GreaterThan"
  },
  "Goal": {
    "Interval": {
      "RollingInterval": {
        "Duration": 1,
        "DurationUnit": "DAY"
      }
    },
    "AttainmentGoal": 99.9,
    "WarningThreshold": 60.0
  }
}' --no-cli-pager --region ap-southeast-1
+ output='
Parameter validation failed:
Invalid type for parameter SliConfig.SliMetricConfig.KeyAttributes, value: None, type: <class '\''NoneType'\''>, valid types: <class '\''dict'\''>'
+ err=252
+ grep InvalidParameterValue
+ echo '
Parameter validation failed:
Invalid type for parameter SliConfig.SliMetricConfig.KeyAttributes, value: None, type: <class '\''NoneType'\''>, valid types: <class '\''dict'\''>'
+ break
+ check_if_loop_failed_and_exit 252 'There was an error creating an SLO - GetOwner99Availability, exiting'
+ '[' 252 -ne 0 ']'
+ echo There was an error creating an SLO - GetOwner99Availability, exiting
There was an error creating an SLO - GetOwner99Availability, exiting
+ exit 1

For some reason, the CloudWatch SLO feature is not able to recognise the EKS cluster as a service.

EKS: image

CloudWatch:

image

Someone else reported this issue back in April. Has this repo been tested recently?

Thanks

pxaws commented 6 days ago

I noticed that you are using this command: 

Admin:~/environment/application-signals-demo/scripts/eks/appsignals/one-step (main) $ ../create-slo.sh unicorn-pet-rentals ap-southeast-1

I don't think this will work by itself. The correct way is to follow this: https://github.com/aws-observability/application-signals-demo?tab=readme-ov-file#deploy-via-shell-scripts by running the whole script.

str3tch commented 6 days ago

Here is the full output from the proper script run:

Admin:~/environment/application-signals-demo/scripts/eks/appsignals/one-step (main) $  ./setup.sh unicorn-pet-rentals ap-southeast-1
AWS CLI version is acceptable, continuing...
2024-10-16 22:32:23 [ℹ]  eksctl version 0.192.0
2024-10-16 22:32:23 [ℹ]  using region ap-southeast-1
2024-10-16 22:32:23 [ℹ]  setting availability zones to [ap-southeast-1a ap-southeast-1b ap-southeast-1c]
2024-10-16 22:32:23 [ℹ]  subnets for ap-southeast-1a - public:192.168.0.0/19 private:192.168.96.0/19
2024-10-16 22:32:23 [ℹ]  subnets for ap-southeast-1b - public:192.168.32.0/19 private:192.168.128.0/19
2024-10-16 22:32:23 [ℹ]  subnets for ap-southeast-1c - public:192.168.64.0/19 private:192.168.160.0/19
2024-10-16 22:32:23 [ℹ]  nodegroup "ng-97f3c11a" will use "" [AmazonLinux2/1.30]
2024-10-16 22:32:23 [ℹ]  using Kubernetes version 1.30
2024-10-16 22:32:23 [ℹ]  creating EKS cluster "unicorn-pet-rentals" in "ap-southeast-1" region with managed nodes
2024-10-16 22:32:23 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial managed nodegroup
2024-10-16 22:32:23 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-southeast-1 --cluster=unicorn-pet-rentals'
2024-10-16 22:32:23 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "unicorn-pet-rentals" in "ap-southeast-1"
2024-10-16 22:32:23 [ℹ]  CloudWatch logging will not be enabled for cluster "unicorn-pet-rentals" in "ap-southeast-1"
2024-10-16 22:32:23 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-southeast-1 --cluster=unicorn-pet-rentals'
2024-10-16 22:32:23 [ℹ]  default addons vpc-cni, kube-proxy, coredns were not specified, will install them as EKS addons
2024-10-16 22:32:23 [ℹ]
2 sequential tasks: { create cluster control plane "unicorn-pet-rentals",
    2 sequential sub-tasks: {
        2 sequential sub-tasks: {
            1 task: { create addons },
            wait for control plane to become ready,
        },
        create managed nodegroup "ng-97f3c11a",
    }
}
2024-10-16 22:32:23 [ℹ]  building cluster stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:32:24 [ℹ]  deploying stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:32:54 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:33:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:34:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:35:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:36:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:37:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:38:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:39:24 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-cluster"
2024-10-16 22:39:25 [!]  recommended policies were found for "vpc-cni" addon, but since OIDC is disabled on the cluster, eksctl cannot configure the requested permissions; the recommended way to provide IAM permissions for "vpc-cni" addon is via pod identity associations; after addon creation is completed, add all recommended policies to the config file, under `addon.PodIdentityAssociations`, and run `eksctl update addon`
2024-10-16 22:39:25 [ℹ]  creating addon
2024-10-16 22:39:25 [ℹ]  successfully created addon
2024-10-16 22:39:25 [ℹ]  creating addon
2024-10-16 22:39:25 [ℹ]  successfully created addon
2024-10-16 22:39:26 [ℹ]  creating addon
2024-10-16 22:39:26 [ℹ]  successfully created addon
 2024-10-16 22:41:26 [ℹ]  building managed nodegroup stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:41:26 [ℹ]  deploying stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:41:26 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:41:56 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:42:48 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:44:16 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-nodegroup-ng-97f3c11a"
2024-10-16 22:44:16 [ℹ]  waiting for the control plane to become ready
2024-10-16 22:44:17 [✔]  saved kubeconfig as "/home/ec2-user/.kube/config"
2024-10-16 22:44:17 [ℹ]  no tasks
2024-10-16 22:44:17 [✔]  all EKS cluster resources for "unicorn-pet-rentals" have been created
2024-10-16 22:44:17 [✔]  created 0 nodegroup(s) in cluster "unicorn-pet-rentals"
2024-10-16 22:44:17 [ℹ]  nodegroup "ng-97f3c11a" has 2 node(s)
2024-10-16 22:44:17 [ℹ]  node "ip-192-168-50-157.ap-southeast-1.compute.internal" is ready
2024-10-16 22:44:17 [ℹ]  node "ip-192-168-6-111.ap-southeast-1.compute.internal" is ready
2024-10-16 22:44:17 [ℹ]  waiting for at least 2 node(s) to become ready in "ng-97f3c11a"
2024-10-16 22:44:17 [ℹ]  nodegroup "ng-97f3c11a" has 2 node(s)
2024-10-16 22:44:17 [ℹ]  node "ip-192-168-50-157.ap-southeast-1.compute.internal" is ready
2024-10-16 22:44:17 [ℹ]  node "ip-192-168-6-111.ap-southeast-1.compute.internal" is ready
2024-10-16 22:44:17 [✔]  created 1 managed nodegroup(s) in cluster "unicorn-pet-rentals"
2024-10-16 22:44:18 [ℹ]  kubectl command should work with "/home/ec2-user/.kube/config", try 'kubectl get nodes'
2024-10-16 22:44:18 [✔]  EKS cluster "unicorn-pet-rentals" in "ap-southeast-1" region is ready
Cluster unicorn-pet-rentals has been created successfully in region ap-southeast-1
Enabling Application Signals for EKS Cluster unicorn-pet-rentals in ap-southeast-1 for namespace default

An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForCloudWatchApplicationSignals has been taken in this account, please try a different suffix.
2024-10-16 22:44:21 [ℹ]  will create IAM Open ID Connect provider for cluster "unicorn-pet-rentals" in "ap-southeast-1"
2024-10-16 22:44:21 [✔]  created IAM Open ID Connect provider for cluster "unicorn-pet-rentals" in "ap-southeast-1"
Creating ServiceAccount
2024-10-16 22:44:22 [ℹ]  1 iamserviceaccount (amazon-cloudwatch/cloudwatch-agent) was included (based on the include/exclude rules)
2024-10-16 22:44:22 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
2024-10-16 22:44:22 [ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "amazon-cloudwatch/cloudwatch-agent",
        create serviceaccount "amazon-cloudwatch/cloudwatch-agent",
    } }2024-10-16 22:44:22 [ℹ]  building iamserviceaccount stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2024-10-16 22:44:22 [ℹ]  deploying stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2024-10-16 22:44:22 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2024-10-16 22:44:52 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-amazon-cloudwatch-cloudwatch-agent"
2024-10-16 22:44:52 [ℹ]  created namespace "amazon-cloudwatch"
2024-10-16 22:44:52 [ℹ]  created serviceaccount "amazon-cloudwatch/cloudwatch-agent"
Checking amazon-cloudwatch-observability add-on

An error occurred (ResourceNotFoundException) when calling the DescribeAddon operation: No addon: amazon-cloudwatch-observability found in cluster: unicorn-pet-rentals
Installing amazon-cloudwatch-observability add-on
{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "unicorn-pet-rentals",
        "status": "CREATING",
        "addonVersion": "v2.1.3-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:ap-southeast-1:267269458124:addon/unicorn-pet-rentals/amazon-cloudwatch-observability/16c94bc0-b635-46e5-5fb3-e4199bb3f481",
        "createdAt": "2024-10-16T22:44:54.686000+00:00",
        "modifiedAt": "2024-10-16T22:44:54.709000+00:00",
        "tags": {}
    }
}
Current status: CREATING
Waiting for addon to become ACTIVE...
Current status: CREATING
Waiting for addon to become ACTIVE...
EKS amazon-cloudwatch-observability add-on is now ACTIVE
Adding EBS CSI Controller for EKS Cluster unicorn-pet-rentals in ap-southeast-1 for namespace default
2024-10-16 22:45:39 [ℹ]  1 existing iamserviceaccount(s) (amazon-cloudwatch/cloudwatch-agent) will be excluded
2024-10-16 22:45:39 [ℹ]  1 iamserviceaccount (kube-system/ebs-csi-controller-sa) was included (based on the include/exclude rules)
2024-10-16 22:45:39 [!]  serviceaccounts in Kubernetes will not be created or modified, since the option --role-only is used
2024-10-16 22:45:39 [ℹ]  1 task: { create IAM role for serviceaccount "kube-system/ebs-csi-controller-sa" }
2024-10-16 22:45:39 [ℹ]  building iamserviceaccount stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
2024-10-16 22:45:39 [ℹ]  deploying stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
2024-10-16 22:45:39 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
2024-10-16 22:46:09 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa"
Checking aws-ebs-csi-driver add-on

An error occurred (ResourceNotFoundException) when calling the DescribeAddon operation: No addon: aws-ebs-csi-driver found in cluster: unicorn-pet-rentals
Installing aws-ebs-csi-driver add-on
{
    "addon": {
        "addonName": "aws-ebs-csi-driver",
        "clusterName": "unicorn-pet-rentals",
        "status": "CREATING",
        "addonVersion": "v1.35.0-eksbuild.1",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:ap-southeast-1:267269458124:addon/unicorn-pet-rentals/aws-ebs-csi-driver/b2c94bc1-4c8e-4396-93c4-038712d2cc10",
        "createdAt": "2024-10-16T22:46:11.636000+00:00",
        "modifiedAt": "2024-10-16T22:46:11.648000+00:00",
        "serviceAccountRoleArn": "arn:aws:iam::267269458124:role/AmazonEKS_EBS_CSI_DriverRole_ap-southeast-1",
        "tags": {}
    }
}
Current status: CREATING
Waiting for addon to become ACTIVE...
Current status: CREATING
Waiting for addon to become ACTIVE...
Current status: CREATING
Waiting for addon to become ACTIVE...
EKS aws-ebs-csi-driver add-on is now ACTIVE
Creating ServiceAccount
2024-10-16 22:47:16 [ℹ]  2 existing iamserviceaccount(s) (amazon-cloudwatch/cloudwatch-agent,kube-system/ebs-csi-controller-sa) will be excluded
2024-10-16 22:47:16 [ℹ]  1 iamserviceaccount (default/visits-service-account) was included (based on the include/exclude rules)
2024-10-16 22:47:16 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
2024-10-16 22:47:16 [ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "default/visits-service-account",
        create serviceaccount "default/visits-service-account",
    } }2024-10-16 22:47:16 [ℹ]  building iamserviceaccount stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-default-visits-service-account"
2024-10-16 22:47:16 [ℹ]  deploying stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-default-visits-service-account"
2024-10-16 22:47:16 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-default-visits-service-account"
2024-10-16 22:47:47 [ℹ]  waiting for CloudFormation stack "eksctl-unicorn-pet-rentals-addon-iamserviceaccount-default-visits-service-account"
2024-10-16 22:47:47 [ℹ]  created serviceaccount "default/visits-service-account"
deployment.apps/db created
persistentvolumeclaim/ebs-claim created
service/db created
storageclass.storage.k8s.io/ebs-sc created
deployment.apps/mongodb created
persistentvolumeclaim/mongodb-pvc-data created
service/mongodb created
storageclass.storage.k8s.io/ebs-sc-mongodb created
deployment.apps/admin-server-java created
service/admin-server-java created
deployment.apps/pet-clinic-frontend-java created
service/pet-clinic-frontend-java created
deployment.apps/billing-service-python created
service/billing-service-python created
deployment.apps/config-server created
service/config-server created
deployment.apps/customers-service-java created
service/customers-service-java created
persistentvolumeclaim/data created
deployment.apps/discovery-server created
service/discovery-server created
deployment.apps/insurance-service-python created
service/insurance-service-python created
deployment.apps/nutrition-service-nodejs created
service/nutrition-service-nodejs created
deployment.apps/payment-service-dotnet created
service/payment-service-dotnet created
deployment.apps/vets-service-java created
service/vets-service-java created
deployment.apps/visits-service-java created
service/visits-service-java created
namespace/ingress-nginx created
namespace/ingress-nginx unchanged
service/ingress-nginx created
deployment.apps/default-http-backend created
service/default-http-backend created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
ingress.networking.k8s.io/petclinic-nginx-ingress created
Visit the following URL to see the sample app running: a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
deployment.apps/traffic-generator created
Started the traffic generator to send traffic to http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
REGION is ap-southeast-1
OPERATION is create
ACCOUNT_ID is 267269458124
Setting up the canary execution role
Start creating IAM role CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1
{
    "Role": {
        "Path": "/service-role/",
        "RoleName": "CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
        "RoleId": "AROAT4OUAJDGDXO3ZB2LT",
        "Arn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
        "CreateDate": "2024-10-16T22:50:06+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}
creating IAM policy
{
    "Policy": {
        "PolicyName": "CloudWatchSyntheticsPolicy-PetClinic-267269458124-ap-southeast-1",
        "PolicyId": "ANPAT4OUAJDGME2DGMC46",
        "Arn": "arn:aws:iam::267269458124:policy/service-role/CloudWatchSyntheticsPolicy-PetClinic-267269458124-ap-southeast-1",
        "Path": "/service-role/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2024-10-16T22:50:10+00:00",
        "UpdateDate": "2024-10-16T22:50:10+00:00"
    }
}
Setting up S3 code buckets
Creating S3 code bucket aws-synthetics-code-petclinic-267269458124-ap-southeast-1
make_bucket: aws-synthetics-code-petclinic-267269458124-ap-southeast-1
Creating S3 artifact bucket cw-syn-results-petclinic-267269458124-ap-southeast-1
make_bucket: cw-syn-results-petclinic-267269458124-ap-southeast-1
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-add-visit.js (deflated 70%)
Uploading canary script for canary pc-add-visit to S3
{
  "ETag": "\"285e58965428ce04151e9648a3266973\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-add-visit
{
  "Canary": {
    "Id": "2392c242-a52d-4922-b15c-6a8e14475e2b",
    "Name": "pc-add-visit",
    "Code": {
      "Handler": "pc-add-visit.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:22.884000+00:00",
      "LastModified": "2024-10-16T22:50:22.884000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-create-owners.js (deflated 81%)
Uploading canary script for canary pc-create-owners to S3
{
  "ETag": "\"600e43658e56bd478494025175d760b1\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-create-owners
{
  "Canary": {
    "Id": "e6034de8-b9b7-466a-a394-2ec77dde9b0d",
    "Name": "pc-create-owners",
    "Code": {
      "Handler": "pc-create-owners.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:28.222000+00:00",
      "LastModified": "2024-10-16T22:50:28.222000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-visit-pet.js (deflated 59%)
Uploading canary script for canary pc-visit-pet to S3
{
  "ETag": "\"a72c792ba1139486969dc38677935d79\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-visit-pet
{
  "Canary": {
    "Id": "344d77e1-0744-467f-b892-77f9173bf10b",
    "Name": "pc-visit-pet",
    "Code": {
      "Handler": "pc-visit-pet.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:33.561000+00:00",
      "LastModified": "2024-10-16T22:50:33.561000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-visit-vet.js (deflated 55%)
Uploading canary script for canary pc-visit-vet to S3
{
  "ETag": "\"ad92f45ac0255d0070d2c5ee21f1bf71\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-visit-vet
{
  "Canary": {
    "Id": "3b32c069-49db-43b5-8e2b-148aaa902a01",
    "Name": "pc-visit-vet",
    "Code": {
      "Handler": "pc-visit-vet.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:38.900000+00:00",
      "LastModified": "2024-10-16T22:50:38.900000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pet-clinic-traffic.js (deflated 64%)
Uploading canary script for canary pet-clinic-traffic to S3
{
  "ETag": "\"a505a6a3688ae3ef43b26a7bb5f6eb40\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pet-clinic-traffic
{
  "Canary": {
    "Id": "020ff953-9ebb-41d7-932e-d732579c4e0c",
    "Name": "pet-clinic-traffic",
    "Code": {
      "Handler": "pet-clinic-traffic.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:44.310000+00:00",
      "LastModified": "2024-10-16T22:50:44.310000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-visit-insurances.js (deflated 55%)
Uploading canary script for canary pc-visit-insurances to S3
{
  "ETag": "\"0269a6959fc673ea350fdf2c13516a7b\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-visit-insurances
{
  "Canary": {
    "Id": "7cf415e9-f450-48b7-84b2-5f149baf747e",
    "Name": "pc-visit-insurances",
    "Code": {
      "Handler": "pc-visit-insurances.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:49.627000+00:00",
      "LastModified": "2024-10-16T22:50:49.627000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/pc-visit-billings.js (deflated 55%)
Uploading canary script for canary pc-visit-billings to S3
{
  "ETag": "\"d568f69a2e78ab6bd00387602fd1221f\"",
  "ServerSideEncryption": "AES256"
}
ENDPOINT is http://a38d5ea1dc1c44c30b2ae71b8bca6e81-1124858756.ap-southeast-1.elb.amazonaws.com
Creating/updating canary pc-visit-billings
{
  "Canary": {
    "Id": "6cda4635-ea89-4d43-a69c-ff071366c4cc",
    "Name": "pc-visit-billings",
    "Code": {
      "Handler": "pc-visit-billings.handler"
    },
    "ExecutionRoleArn": "arn:aws:iam::267269458124:role/service-role/CloudWatchSyntheticsRole-PetClinic-267269458124-ap-southeast-1",
    "Schedule": {
      "Expression": "rate(1 minute)",
      "DurationInSeconds": 0
    },
    "RunConfig": {
      "TimeoutInSeconds": 60,
      "MemoryInMB": 1000,
      "ActiveTracing": true
    },
    "SuccessRetentionPeriodInDays": 31,
    "FailureRetentionPeriodInDays": 31,
    "Status": {
      "State": "CREATING",
      "StateReasonCode": "CREATE_PENDING"
    },
    "Timeline": {
      "Created": "2024-10-16T22:50:54.920000+00:00",
      "LastModified": "2024-10-16T22:50:54.920000+00:00"
    },
    "ArtifactS3Location": "cw-syn-results-petclinic-267269458124-ap-southeast-1",
    "RuntimeVersion": "syn-nodejs-puppeteer-6.2",
    "Tags": {}
  }
}
Waiting a minute for canaries to finish creating or updating.
Done waiting. Starting canaries.
++ aws synthetics get-canary --name pc-add-visit --region ap-southeast-1
++ set +x
Starting canary pc-add-visit
++ aws synthetics get-canary --name pc-create-owners --region ap-southeast-1
++ set +x
Starting canary pc-create-owners
++ aws synthetics get-canary --name pc-visit-pet --region ap-southeast-1
++ set +x
Starting canary pc-visit-pet
++ aws synthetics get-canary --name pc-visit-vet --region ap-southeast-1
++ set +x
Starting canary pc-visit-vet
++ aws synthetics get-canary --name pet-clinic-traffic --region ap-southeast-1
++ set +x
Starting canary pet-clinic-traffic
++ aws synthetics get-canary --name pc-visit-insurances --region ap-southeast-1
++ set +x
Starting canary pc-visit-insurances
++ aws synthetics get-canary --name pc-visit-billings --region ap-southeast-1
++ set +x
Starting canary pc-visit-billings
Wait 10 minutes for canaries to generate traffic
Creating Service Level Objectives
null
There was an error creating an SLO - GetOwner99Availability, exiting
There was an error creating the SLOs, exiting
str3tch commented 6 days ago

I can see that when it tried to create the service linked role, it already existed and failed, but I do not think this affected the rest of the steps in the script.

An error occurred (InvalidInput) when calling the CreateServiceLinkedRole operation: Service role name AWSServiceRoleForCloudWatchApplicationSignals has been taken in this account, please try a different suffix.

Any thoughts?

pxaws commented 6 days ago

I think this is a benign warning which shouldn't matter. I suspect the issue might be services are not running correctly in the EKS cluster. Could you run kubectl get pod and post the output here? (maybe some pods are not running correctly)