Version starting 22.1.0 cannot find existing AWS connector

[vtstanescu]

Hello,

We have a production workload of CVO on AWS deployed & managed using this provider. Recently we run Terraform plan (TF version 1.0.3) through our pipeline which picks the latest netapp-cloudmanager provider version. We noticed the provider is not able to find the existing AWS connectors and after verifying they are still alive & healthy, we started pinning the provider version to older ones to see if we can find the problem there. 22.2.0 & 22.1.0 have this problem, while 21.12.0 works fine (sees the connectors and doesn't try to recreate them).

[wenjun666]

Hi,

I just used 22.1.0 and 22.2.0 to create AWS connector, but couldn't reproduce this error.

[vtstanescu]

Hi, Justin!

I'll try replicating it, next week, with a new connector as I cannot risk affecting the existing ones and get back with TRACE logs from Terraform.

Meanwhile, it's worth noting the last provider version which updated those connectors was 21.x, they were also created with 21.x. I'll try that scenario as well as creating & updating with 22.x directly.

[carchi8py]

@MindBreakerVictor Did you get a chance to try again?

[vtstanescu]

Hello,

I've just done so this morning and found a weird thing, this issue (NetApp connector not being seen by Terraform and thus planning to create one - pipeline will fail downstream as the client_id cannot be changed on volume resource) is only happening with Terraform (and the providers) on linux_amd64 architecture, which is the architecture of our containers running CI processes. On my MBP, darwin_amd64 architecture, this is not happening, Terraform is showing no infra changes required as expected.

I'm using the same Terraform version and providers' versions in both as you can see the output of terraform version for both:

Let me know if a TRACE log will be helpful here, I can get them for both scenarios. Although they might be big files since we have quite a bit of stuff in there and also hard to redact them.

[alikulov]

Hi. I have the same issue. I have netapp-cloudmanager_connector_aws which was created previously and version 21.9.4 of netapp-cloudmanager provider was used. I've updated provider version in code to 22.1.0 and terraform plan shows that I need to create connector as if it is absent. If I change provider version to 21.12.0 it finds connector.

[alikulov]

@vtstanescu Hi. Did you manage to solve the issue?

[vtstanescu]

I haven't tried further, seems to be at provider level, be it code or build between architectures as in my case this issue occurs with the provider in the Linux architecture, but not occurring on Darwin (macOS)

[alikulov]

I tried to run terraform plan locally on MacOS. It still can't find created connector:

Terraform v1.2.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.75.2
+ provider registry.terraform.io/hashicorp/vault v3.7.0
+ provider registry.terraform.io/netapp/netapp-cloudmanager v22.1.0

@wenjun666 @carchi8py this bug is quite critical for us. We have a couple of CVO clusters in production and we wouldn't like to recreate connectors to use newer versions of the provider.

[wenjun666]

I created a connector in 21.9.4 and then switch to 22.1.0 but didn't get any change if run terraform plan. It's on my local MacOS.

$ terraform version
Terraform v1.0.1
on darwin_amd64
+ provider registry.terraform.io/netapp/netapp-cloudmanager v22.1.0

Your version of Terraform is out of date! The latest version
is 1.2.4. You can update by downloading from https://www.terraform.io/downloads.html

we introduced 'aws_profile' in 22.1.0, 'This is the profile name of the aws credentials file in your home directory, for example,~/.aws/credentials. If not specified, profile named default is used.' Will that be a possible cause?

[alikulov]

Looks like 'aws_profile' is the issue. I've managed to run terraform plan locally on my laptop w/o issues after I defined 'aws_profile'. But the issue still persists on our jenkins nodes because we have them running in AWS and don't use ~/.aws/credentials file at all.

[wenjun666]

@alikulov can you elaborate what's running in AWS?

[alikulov]

We use Jenkins running on EC2 in AWS to run terraform code. And EC2 instances use IAM instance profiles for auth so there is no ~/.aws/credentials file on Jenkins workers.

[wenjun666]

    // Path to the shared credentials file.
    //
    // If empty will look for "AWS_SHARED_CREDENTIALS_FILE" env variable. If the
    // env value is empty will default to current user's home directory.
    // Linux/OSX: "$HOME/.aws/credentials"
    // Windows:   "%USERPROFILE%\.aws\credentials"
    Filename string

    // AWS Profile to extract credentials from the shared credentials file. If empty
    // will default to environment variable "AWS_PROFILE" or "default" if
    // environment variable is also not set.
    Profile string

The filename is currently always set to default, my propose fix is to make the file name an option for user to specify.

[alikulov]

I'm not sure that your propose will fix the issue in our case. Because you still assume that there will be a credentials file. But in case of EC2 credentials are acquired from the EC2 instance metadata service.

For applications, AWS CLI, and Tools for Windows PowerShell commands that run on the instance, you do not have to explicitly get the temporary security credentials—the AWS SDKs, AWS CLI, and Tools for Windows PowerShell automatically get the credentials from the EC2 instance metadata service and use them.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

[wenjun666]

okay, This should work:

in 21.9.4, we use sess, err := session.NewSession(), it looks for credentials in follow orders: Environment Variables Shared Credentials file Shared Configuration file (if SharedConfig is enabled) EC2 Instance Metadata (credentials only)

we are going to use that functions again to create session, unless a user specifically provide cred files. In that way, it will search for credentials it self.

[wenjun666]

Fix in v22.8.0. Please reopen if issue still exists.

[alikulov]

Just to confirm. We've tested v22.8.0. The issue is fixed. Thanks