[f5-cloud-failover] Status: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 Retries left: 0

[sd224646]

Do you already have an issue opened with F5 support?

no.

Description

Getting below error on AWS F5 and failover not working [f5-cloud-failover] Status: Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 Retries left: 0

Environment information

For bugs, enter the following information:

• Cloud Failover Extension Version: 1.15.0
• BIG-IP version: 17.1.0.3

Severity Level

2

Severity: <Fill in level: 1 through 5>

Severity level definitions:

1. Severity 1 (Critical) : Defect is causing systems to be offline and/or nonfunctional. immediate attention is required.
2. Severity 2 (High) : Defect is causing major obstruction of system operations.
3. Severity 3 (Medium) : Defect is causing intermittent errors in system operations.
4. Severity 4 (Low) : Defect is causing infrequent interuptions in system operations.
5. Severity 5 (Trival) : Defect is not causing any interuptions to system operations, but none-the-less is a bug.

[mikeshimkus]

Hi @sd224646 This error usually happens when the IAM role/policy isn't assigned to the BIG-IP instances. Can you verify that a role with the correct policy permissions is assigned per https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#create-and-assign-an-iam-role

[sd224646]

Thanks, you are correct, I had created IAM role/policy and missed to attach on BIG-IP instances. Further I can see now failover is working but failover will happen once below errors count goes to 0. Please review and confirm if any finetune required on IAM.

Existing IAM permissions: "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeAddresses", "ec2:DescribeNetworkInterfaces", "ec2:DescribeNetworkInterfaceAttribute", "ec2:DescribeRouteTables", "s3:ListAllMyBuckets", "ec2:AssociateAddress", "ec2:DisassociateAddress", "ec2:AssignPrivateIpAddresses", "ec2:UnassignPrivateIpAddresses", "ec2:ReplaceRoute", "ec2:CreateRoute", "ec2:DeleteRoute", "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface"

error:

Sun, 22 Oct 2023 13:44:27 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 49 Sun, 22 Oct 2023 13:44:32 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 48 Sun, 22 Oct 2023 13:44:37 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 47 Sun, 22 Oct 2023 13:44:42 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 46 Sun, 22 Oct 2023 13:44:47 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 45 Sun, 22 Oct 2023 13:44:52 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. Retries left: 44

[mikeshimkus]

You only have one S3 permission, but seven are required. These are the permissions you will need: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#create-and-assign-an-iam-role

[sd224646]

Below is updated IAM policy with error. its retry for approx. 40 times then will do failover. Note: after 40 retry failover is success. need to know why this is trying 40 retry

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeSubnets", "ec2:DescribeAddresses", "ec2:DescribeNetworkInterfaces", "ec2:DescribeNetworkInterfaceAttribute", "ec2:DescribeRouteTables", "s3:ListAllMyBuckets", "ec2:AssociateAddress", "ec2:DisassociateAddress", "ec2:AssignPrivateIpAddresses", "ec2:UnassignPrivateIpAddresses", "ec2:ReplaceRoute", "ec2:CreateRoute", "ec2:DeleteRoute", "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "logs:CreateLogStream", "logs:CreateLogGroup", "logs:PutLogEvents" ], "Resource": "", "Effect": "Allow" }, { "Action": [ "sts:AssumeRole" ], "Resource": "arn:aws:iam::123456789123:role/F5_cfe_IAM_Role", "Effect": "Allow" }, { "Action": [ "ec2:CreateRoute", "ec2:ReplaceRoute" ], "Resource": [ "arn:aws:ec2:us-east-1:123456789123:route-table/rtb-1111111111111111", "arn:aws:ec2:us-east-1:123456789123:route-table/rtb-2222222222222222", "arn:aws:ec2:us-east-1:123456789123:route-table/rtb-33333333333333333" ], "Condition": { "StringEquals": { "ec2:ResourceTag/f5_cloud_failover_label": "Big-IP-F5" } }, "Effect": "Allow" }, { "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:GetBucketTagging" ], "Resource": "arn:aws:s3:::s3-f5-cfe", "Effect": "Allow" }, { "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::s3-f5-cfe/", "Effect": "Allow" } ] }

Mon, 23 Oct 2023 17:25:20 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action Retries left: 38 Mon, 23 Oct 2023 17:25:25 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action Retries left: 37 . .Will go till retry left 0 . . Mon, 23 Oct 2023 17:28:27 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no permissions boundary allows the ec2:DescribeSubnets action Retries left: 1 Mon, 23 Oct 2023 17:28:32 GMT - finest: [f5-cloud-failover] Status: You are not authorized to perform this operation. User: arn:aws:sts::123456789123:assumed-role/F5_cfe_IAM_Role/i-aaaaaaaaaaaaaaaa is not authorized to perform: ec2:DescribeSubnets because no permissions boundary allows the ec2:DescribeSubnets action Retries left: 0

[mikeshimkus]

The error message is clear, the policy does not have the DescribeSubnets permission. It looks like the resource for that permission in your policy is blank. Check out https://github.com/F5Networks/f5-aws-cloudformation-v2/blob/8afb5e28975f42e70f64d2808735f91bd2663f44/examples/modules/access/access.yaml#L916 for an example from our CloudFormation templates.

[sd224646]

Thanks, I will modify IAM and will come back with upadate.

[sd224646]

Have verified again resource for IAM in your policy is not blank. not sure why "" is not showing i above comment. further I am using () and allow for"ec2:DescribeSubnets". not sure why I am getting error saying You are not authorized to perform this operation. Could you please any working full IAM. my purpose is to replace destination nic in route table.

[sd224646]

ok got it. looking like Github comment section not allow "asterisk". same i can see in my above comments.

[mikeshimkus]

The example shown above is the full working policy: https://clouddocs.f5.com/products/extensions/f5-cloud-failover/latest/userguide/aws.html#iam-role-example-declaration

Your policy looks quite different from what's documented (you do not need sts:AssumeRole, for example), I would create a policy based on the example above and try again.

[sd224646]

No Luck, first statement ""BigIpHighAvailabilityAccessRole"" itself not accepted by IAM. then all "\asterisk" also not accepted.

[mikeshimkus]

That's the policy name, I don't think the editor will allow you to use it in the policy itself. The asterisks need quotes around them, that's a typo.

Can you update your existing policy to match our example?