Agent keeps retrying instance profile credentials instead of SSM role

[JackelynOliveira]

SSM Agent: 3.2.1705.0 Resource type: EC2 instance OS: Ubuntu 18.04.1 LTS (64 bit)

• Configured Default Host Management Configuration with recommended AWSSystemsManagerDefaultEC2InstanceManagementRole role.
• VM had previously been assigned a role (let's call it CUSTOM_EC2_ROLE) that didn't have the UpdateInstanceInformation permission.
• For troubleshooting purposes, added the permission, but now it's been removed.
• SSM agent seems stuck in a loop where it attempts to use the instance profile permissions to connect to SSM, to then use the recommended role.
• Problem is that this managed node is now shown as having lost connection, on Fleet Manager.
• Already tried restarting the agent.

Logs:

2023-10-25 22:14:42 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::ACCOUNT:assumed-role/CUSTOM_EC2_ROLE/INSTANCE_ID is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:REGION:ACCOUNT:instance/INSTANCE_ID because no identity-based policy allows the ssm:UpdateInstanceInformation action
    status code: 400, request id: [...]
2023-10-25 22:16:24 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. RequestId: [...] Error: AccessDeniedException: User: arn:aws:sts::ACCOUNT:assumed-role/CUSTOM_EC2_ROLE/INSTANCE_ID is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:REGION:ACCOUNT:instance/INSTANCE_ID because no identity-based policy allows the ssm:UpdateInstanceInformation action
    status code: 400, request id: [...]
2023-10-25 22:16:24 INFO EC2RoleProvider Successfully connected with Systems Manager role credentials
2023-10-25 22:16:24 INFO [CredentialRefresher] Credentials ready
2023-10-25 22:16:24 INFO [CredentialRefresher] Next credential rotation will be in 29.993814139633333 minutes
2023-10-25 22:46:24 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. RequestId: [...] Error: AccessDeniedException: User: arn:aws:sts::ACCOUNT:assumed-role/CUSTOM_EC2_ROLE/INSTANCE_ID is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:REGION:ACCOUNT:instance/INSTANCE_ID because no identity-based policy allows the ssm:UpdateInstanceInformation action
    status code: 400, request id: [...]
2023-10-25 22:46:24 INFO EC2RoleProvider Successfully connected with Systems Manager role credentials
2023-10-25 22:46:24 INFO [CredentialRefresher] Credentials ready
2023-10-25 22:46:24 INFO [CredentialRefresher] Next credential rotation will be in 29.995949605633335 minutes
2023-10-25 23:16:24 WARN EC2RoleProvider Failed to connect to Systems Manager with instance profile role credentials. Err: retrieved credentials failed to report to ssm. RequestId: [...] Error: AccessDeniedException: User: arn:aws:sts::ACCOUNT:assumed-role/CUSTOM_EC2_ROLE/INSTANCE_ID is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:REGION:ACCOUNT:instance/INSTANCE_ID because no identity-based policy allows the ssm:UpdateInstanceInformation action
    status code: 400, request id: [...]
2023-10-25 23:16:24 INFO EC2RoleProvider Successfully connected with Systems Manager role credentials
2023-10-25 23:16:24 INFO [CredentialRefresher] Credentials ready
2023-10-25 23:16:24 INFO [CredentialRefresher] Next credential rotation will be in 29.997133145116667 minutes

[cjinaws]

The log below shows that the default host management role is being used and that the instance is connected to Systems Manager.

2023-10-25 22:16:24 INFO EC2RoleProvider Successfully connected with Systems Manager role credentials

[sluggard76]

Based on the observation by cjinaws, there are no issues to obtain the credential. Closing the issue. Feel free to reopen or create a new issue if the problem persists.