[BUG] NoCredentialProviders for assuming role in provider

[robert-becker-hs]

What is the bug?

Using the aws_assume_role_arn option in the provider config ends with throwing a NoCredentialProviders error during Terraform apply stage.

I have an active Open Search cluster in AWS, all IAM roles exist.

Terraform

module "es" {
  // This is a local module which creates the OpenSearch cluster in AWS
  source = "./modules/aws_opensearch"
...
}

provider "opensearch" {
  URL = module.es.endpoint
  aws_region  = var.region
  sign_aws_requests  = true
  healthcheck = false
  aws_assume_role_arn = module.es.master_role_arn
  opensearch_version = var.es_version
}

resource "opensearch_role" "admin_role" {
  role_name   = "bot-information-retrieval-admin"
  description = "Admin role"
}

Result

opensearch_role.admin_role: Creating...
╷
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│   For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│ 
│   with opensearch_role.admin_role,
│   on main.tf line 127, in resource "opensearch_role" "admin_role":
│  127: resource "opensearch_role" "admin_role" {

How can one reproduce the bug?

Try using the role assumption for any OpenSearch change and the above error shows up.

What is the expected behaviour?

The OpenSearch role was created successfully using the assumed role.

What is your host/environment?

MacOS 14.5 (M1 Mac)

$ terraform version
Terraform v1.3.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.61.0
+ provider registry.terraform.io/opensearch-project/opensearch v2.3.0

Do you have any additional context?

If I am not using the aws_assume_role_arn property and run Terraform, the apply times out after 5 minutes. I checked that I am able to assume the role module.es.master_role_arn. I also checked similar open issues and tried out several other config settings but without success. Downgrading the module to lower minor version also yielded no results

[gaiksaya]

[Triage] Hi @robert-becker-hs ,

Are you using aws terraform provider or terraform-provider-opensearch to deploy the cluster?

[spr-mweber3]

@gaiksaya Seeing the same issue. We are indeed creating the OpenSearch cluster with the AWS provider. Why is that important? Anything special to consider to make it work?

Example:

Error: NoCredentialProviders: no valid providers in chain. Deprecated.

When trying to do something inside the AWS OpenSearch with the OpenSearch provider like creating an opensearch_user resource it fails with error when using aws_assume_role_arn. It works perfectly fine though, if I set aws_profile in the provider configuration pointing at the exact same role that I try to directly assume with aws_assume_role_arn.

Could it be that you use a depracated AWS SDK in your implementation of the provider and the assuming isn't working (anymore)?

Another thing I came across, not sure though whether it is related (but feels like it), ...

Example: When trying to do something inside the AWS OpenSearch with the OpenSearch provider like creating an opensearch_user and I try via aws_assume_role_arn to switch into a role that I specifically created the attempt to create any resource fails with error Error: elastic: Error 403 (Forbidden). I had debug log switched on and I was able to see that the provider assumed the role, though. It was a different role from that role that created the cluster, but it had AdministratorAccess. So, I assume it should be enough.

But maybe there is some magic happening in AWS OpenSearch like AWS is doing on their EKS clusters (earlier) where the IAM role that created the EKS cluster initially was always granted cluster-admin access inside the Kubernetes cluster.

[prudhvigodithi]

Hey @robert-becker-hs and @spr-mweber3 here is some discussion from past issue https://github.com/opensearch-project/terraform-provider-opensearch/issues/61#issuecomment-1718089241. Also an example https://github.com/rblcoder/terraform-opensearch-samples/blob/main/aws_opensearch_assume_role/main.tf on how to use aws_assume_role_arn. Can you please try and post your thought here? Thanks @getsaurabh02 @rblcoder

[fmlisco]

@prudhvigodithi I have the same issue as already described by @robert-becker-hs and @spr-mweber3, and nothing seems to be effective in fixing it. I went through issue #61, but I didn't find anything helpful.

In short, I can make it work if I use aws_profile, but unfortunately, it doesn't seem to work if I provide aws_assume_role_arn, regardless of the permissions assigned to this role.

Could this issue be maybe related to the fact that I am running Terraform as an SSO-authenticated user?

[prudhvigodithi]

Hey @fmlisco can you try to use aws_assume_role_external_id with aws_assume_role_arn

provider "opensearch" {
  url = "url"
  healthcheck        = "false"
  aws_region          = "region"
  aws_assume_role_arn = "role arn"
  aws_assume_role_external_id = "opensearch-external"
  version_ping_timeout = "10"
}

An example added here https://github.com/rblcoder/terraform-opensearch-samples/blob/main/aws_opensearch_assume_role/main.tf#L10C1-L18C2.

If aws_assume_role_arn dint work with aws_assume_role_external_id, there must be some problem to solve, can you please try and post the results here please?

Thank you

[fmlisco]

I found the solution and would like to share it here in hopes that it can help someone.
I encountered two problems while using an IAM Role to perform the following tasks:

• I couldn't create a user in OpenSearch.
• I couldn't use an S3 bucket to register a repository for my manual snapshots.

1) To create a user, I had to go to AWS Console -> OpenSearch -> Security Configuration -> Fine-grained access control -> Set IAM ARN as master user -> [the role ARN I am using]. After that I was able to perform any task inside Opensearch.

2) To register a repository, I had to add before the "iam:PassRole" permission to my AWS IAM role.