Closed nalshamaajc closed 7 months ago
Is this the right way to use terraform cloud as a backend?
I switched to using Github as the source of the module while using terraform cloud workspaces as a backend. I passed the token credentials in a .terraformrc
secret.
Namespace: default
Status:
At Provider:
Conditions:
Last Transition Time: 2023-11-24T18:02:24Z
Message: connect failed: cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip
Reason: ReconcileError
Status: False
Type: Synced
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning CannotConnectToProvider 9s managed/workspace.tf.upbound.io cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip
$ echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip
Error: Required token could not be found
Run the following command to generate a token for app.terraform.io:
terraform login
Updated ProviderConfig *
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: terraform
spec:
credentials:
- filename: .terraformrc
source: Secret
secretRef:
namespace: crossplane-system
name: terraformrc
key: .terraformrc
- filename: .git-credentials
source: Secret
secretRef:
namespace: crossplane-system
name: git-credentials
key: .git-credentials
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
backend "remote" {
organization = "ORG"
hostname = "app.terraform.io"
workspaces {
name = "dev-us-west-2-p01-crossplane"
}
}
}
provider "aws" {
region = us-west-2
assume_role {
role_arn = format("arn:aws:iam::%s:role/crossplane", xxxxxxxxxxxxx)
}
default_tags {
tags = {
Source = format("ORG/infra/terraform/%s/%s/%s/foundation", dev, us-west-2, p01-crossplane)
Team = "devops"
CostOrg = "engineering"
ProductLine = "shared"
CrossplaneResource = "True"
Env = dev
}
}
}
I never tested terraform cloud integration with provider-terraform, but it looks like we need to propagate the token in some supported manner, e.g. https://developer.hashicorp.com/terraform/cli/config/config-file#environment-variable-credentials
Thanks @ytsarev
I managed to make it work by adding the token in the providerconfig
, but it seems that terraform workspaces as a backend are not supported right?
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
backend "remote" {
organization = "ORG"
hostname = "app.terraform.io"
token = "XXXXXXXXXXXXXXTOKENXXXXXXXXXXXXXXX"
workspaces {
name = "p01-crossplane-foundation"
}
}
Type Reason Age From Message
---- ------ ---- ---- -------
Warning CannotConnectToProvider 3m56s (x45 over 54m) managed/workspace.tf.upbound.io cannot select Terraform workspace: Terraform encountered an error. Summary: . To see the full error run: echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip
$echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip
Failed to get configured named states: workspaces not supported
Everything that is supported by the standard open source terraform CLI(current version included https://github.com/upbound/provider-terraform/blob/main/cluster/images/provider-terraform/Dockerfile#L6) , should be supported as well.
I would investigate standard terraform issues like https://github.com/hashicorp/terraform/issues/23076#issuecomment-541973720 to proceed.
@ytsarev the Cloud
block should be supported, but even after doing more modifications to narrow it down I still got the below result workspaces not supported
. could I be missing something here? Or is it that workspaces as a state backend are not supported.
PS: the workspace was already created on terraform cloud
spec:
credentials:
- filename: .git-credentials
source: Secret
secretRef:
namespace: crossplane-system
name: git-credentials
key: .git-credentials
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
cloud {
organization = "ORG"
hostname = "app.terraform.io"
token = "XXXXXTOKENXXXX"
workspaces {
name = "crossplane-workspace"
}
}
}
I think this this comment is related to what you posted earlier but gives a bit more context.
Next I tried using workspacetags
and I'm stuck at making it use the actual workspace instead of the defualt
. Where I'm trying to pass the value of the workspace as an environment variable TF_WORKSPACE
, but I don't think the Workspace object supports this.
ProviderConfig
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
cloud {
organization = "ORG"
hostname = "app.terraform.io"
token = "XXXXXXXXXXXXXXXXXXXXTOKENXXXXXXXXXXXXXXXXXXXXXXX"
workspaces {
tags = ["tag1", "tag2", "tag3", "tag4", "tag5"]
}
}
}
Error Message
Status:
At Provider:
Conditions:
Last Transition Time: 2023-11-26T11:08:48Z
Message: connect failed: cannot initialize Terraform configuration: Terraform encountered an error. Summary: Currently selected workspace "default" does not exist. To see the full error run: echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
Reason: ReconcileError
Status: False
Type: Synced
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning CannotObserveExternalResource 97s managed/workspace.tf.upbound.io cannot diff (i.e. plan) Terraform configuration: Terraform encountered an error. Summary: Variables not allowed. To see the full error run: echo "H4sIAAAAAAAA/7zQQWrDMBCF4b1O8S6gEDuLgMDLXqHbMrFfWhdZY2Zkh96+VBSyLhS/A8z/MeHFTC3hVWyWW6ajaIXkrA9OIQBaMJq6r1kK42q6zxMtVprJXW2J51O972KOPBeiSwFAl2B8n7UAAzaPD3qNfQjPyiJfrXQjNueEDxpP4b81fdP0CSw7fjZg4n6849Icl4TV9JNjxYD13MXnieNJ10a6JgBj3rzS3oosxO/ao+Jfkd8AAAD//wEAAP///MTLMk4CAAA=" | base64 -d | gunzip
Warning CannotConnectToProvider 29s (x5 over 89s) managed/workspace.tf.upbound.io cannot initialize Terraform configuration: Terraform encountered an error. Summary: Currently selected workspace "default" does not exist. To see the full error run: echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
There are some problems with the CLI configuration:
╷
│ Warning: Unable to open CLI configuration file
│
│ The CLI configuration file at "./.terraformrc" does not exist.
╵
Error: Currently selected workspace "default" does not exist
@ytsarev I tried to use a S3 bucket as the backend and it seems that the provider wasn't using the role I passed to the runtimeConfig
noting that I tried the same configuration for the provider-aws-ec2
and it was able to use the IRSA arn I passed and create the needed aws resources.
ProviderConfig
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
backend "s3" {
bucket = "bucket-dev-terraform-state"
key = "crossplane/foundation/foundation.tfstate"
region = "us-west-2"
dynamodb_table = "crossplane-terraform-lock"
}
Provider / DeploymentRuntimeConfig
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
name: terraform-runtimeconfig
spec:
serviceAccountTemplate:
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXX:role/crossplane
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-terraform
spec:
package: xpkg.upbound.io/upbound/provider-terraform:v0.11.0
runtimeConfigRef:
name: terraform-runtimeconfig
Yet I still get this ERROR
Status:
At Provider:
Conditions:
Last Transition Time: 2023-11-26T13:23:52Z
Message: connect failed: cannot initialize Terraform configuration: Terraform encountered an error. Summary: error configuring S3 Backend: no valid credential sources for S3 Backend found.. To see the full error run: echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip
Reason: ReconcileError
Status: False
Type: Synced
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning CannotConnectToProvider 54s (x4 over 4m13s) managed/workspace.tf.upbound.io (combined from similar events): cannot initialize Terraform configuration: Terraform encountered an error. Summary: error configuring S3 Backend: no valid credential sources for S3 Backend found.. To see the full error run: echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip
echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip
Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
Please see https://www.terraform.io/docs/language/settings/backends/s3.html
for more information about providing credentials.
Error: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 7dfe208e-3570-476e-b32d-9f610b772822
PS: IRSA works for provider-aws-ec2
I tried using the kuberenetes as a backend. Yet still got an error on assuming the needed role.
spec:
credentials:
- filename: aws.json
source: None
- filename: .terraformrc # use exactly this filename by convention
source: Secret
secretRef:
namespace: crossplane-system #upbound-system
name: terraformrc
key: .terraformrc
- filename: .git-credentials # use exactly this filename
source: Secret
secretRef:
namespace: crossplane-system
name: git-credentials
key: .git-credentials
configuration: |
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4"
}
}
backend "kubernetes" {
secret_suffix = "providerconfig-default"
namespace = "crossplane-system"
in_cluster_config = true
}
}
I validated that the ServiceAccount user is assigned to the right role-ARN
kubectl describe sa provider-terraform-bb61cbfc5270 -n crossplane-system
Name: provider-terraform-bb61cbfc5270
Namespace: crossplane-system
Labels: <none>
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/crossplane
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
I checked the OIDC trust policy and it was valid, I also tried creating a resource using the provider-aws-ec2
and it was created successfully. Below is the aws policy condition.
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringLike": {
"oidc.eks.us-west-2.amazonaws.com/id/044A643CAAABC42964BFB2DD152810D6:sub": [
"system:serviceaccount:crossplane-system:provider-aws-*",
"system:serviceaccount:crossplane-system:provider-terraform" # Also tested provider-terraform-*
]
}
echo "H4sIAAAAAAAA/4yTMW/bSBCFe/6KB1d3B1EiLYmithPOKVwkCGwFKYIUK+7IXIDc2cwsbST588HKUhwYLlKwGXK+NzPvsXgnwmLQcTj6h0l8eMCeROyRZcTu8z0+Cj96R2Jwu3uPOx4I/1gJxj6p8XY0pq2qZbuq22W9qY3wQItOWDUONlDpRe2/6GwInHAgWNVpJDcvin1PQrD5QZjGAwn4iMiq/jAQOjspaS6l3itKpJ4wsiZ0PI4ccqcpgP+w7wmdkKOQvB0Uk5KDD2BxJEh81jwB8ngnTR8e7eDdmwDHyNP29pFgYxSO4m0iRJLRq3oOb1B/k04Su7sP8HrCPOsU+ZTnW3MksSljKBdwv7832J1w+bwz9ClFhZBGDkqXr5JNk/7PjgxW1XKGO/o2kabbG4PrqrHr7aYp26Zx5cq129KututyvenW20NzXTeum8FGf2btuo5Ubyh4cgafNLt78VSTvvL07Fn5lreLelPVVb2ul9tq3TZNs63ay+Z2Sj2L/0Eu3yuS5EwZZIGXbcEhb8qTdPQyxN8FqygK4MmnHvEc0i9XQg9ek3yfp0uK554XvdXedyxxYZ/06uusQNb9A3cBlM8/wjwdMfhAqJtZztLlNa5yf85d3ZhXVfwsil8AAAD//wEAAP//xnzGSE8DAAA=" | base64 -d | gunzip
Error: configuring Terraform AWS Provider: IAM Role (arn:aws:iam::xxxxxxxxxxx:role/crossplane) cannot be assumed.
There are a number of possible causes of this - the most common are:
* The credentials used in order to assume the role are invalid
* The credentials do not have appropriate permission to assume the role
* The role ARN is not valid
AWS Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 206a5976-866d-4d89-a495-57c59b6216dc, api error AccessDenied: User: arn:aws:sts::xxxxxxxxxxx:assumed-role/crossplane/1701015139058666908 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/crossplane
with provider["registry.terraform.io/hashicorp/aws"],
on crossplane-provider-config.tf line 16, in provider "aws":
16: provider "aws" {
It feels like something is not right, the docs here mentioned that the process is similar to what we have in provider-aws. But none of the source
options provided there work to force the terraform provider to use a specific role-arn instead of a file, secret, or configMap.
Have you verified that the AWS environment variables are set in the pod as a result of the annotation?
@bobh66 They are
$ kubectl exec -it provider-terraform-bb61cbfc5270-867bb7f685-bbs9h -n crossplane-system -- printenv | grep -i aws
AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=us-west-2
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxx:role/crossplane
AWS_REGION=us-west-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
I tested older version v0.7.0
and v0.2.0
. So far all the tests failed.
It only worked after I passed a user token and used AWS s3 as a backend. This is not ideal in my case since the project that I'm working on uses IRSA and terraform cloud as a backend.
ProviderConfig
credentials:
- filename: .terraformrc
source: Secret
secretRef:
namespace: crossplane-system
name: terraformrc
key: .terraformrc
- filename: aws-credentials # <---------- AWS Credentials passed as secrets
source: Secret
secretRef:
name: aws-credentials
namespace: crossplane-system
key: credentials
- filename: .git-credentials
source: Secret
secretRef:
namespace: crossplane-system
name: git-credentials
key: .git-credentials
configuration: |
provider "aws" {
region = "us-west-2"
shared_credentials_files = ["aws-credentials"] # <------------ Credentials file instead of role-arn
it sounds like a problem with the Assume Role configuration - are you sure that the provider-terraform service account name is right? You might add "" to the end of the service account name in the AssumeRole definition so that any provider-terraform service account can assume the role.
Yeah the name is right @bobh66, seems I figured it out. Looks like the issue was caused by the assume_role
block. Removing it solved the problem, yet I'm still curious why this block was causing the issue with the role used here being the same for IRSA.
provider "aws" {
region = "us-west-2"
assume_role {
role_arn = "arn:aws:iam::xxxxxxxx:role/crossplane"
}
I have seen problems with a role trying to assume itself - that used to be allowed by default but it was changed some time ago to require specific provisioning to allow it to work.
Glad you got it working!
Thank you @bobh66
I hope I get the same luck with the terraformCloud backend issue.
More testing was done and I noticed that the workspace was creating a new workspace on other than the one I provided which matched the external-name
annotation in the workspace object.
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
annotations:
crossplane.io/external-name: "crossplane_test"
meta.upbound.io/example-id: tf/v1beta1/workspace
I changed the value of the external-name
annotation to match the workspace I created and the "workspace not supported" error mentioned in this comment was gone.
I was now left with a different error which was complaining about the absence of a state file in this backend.
echo "H4sIAAAAAAAA/0yOMU7FQAxE+73Cb4YD8AXiHhRAl8psvFlLWTt4vYq4PUoUpF+48cy8mdv09treDT0oGEVWxk4dxYbOTyl9nu9GSgs31kC21kjnDuefIc6gh+gdH0MRVfq/L4mCMItzDvNf7JWd8cXuVMwbKnV8Myt8KMwxOiMq4/mCrrSkMGwmGqdwcRF2NG+cpUi+JqyWKcT0fpte2nHpDwAA//8BAAD//yMXozveAAAA" | base64 -d | gunzip
No state file was found!
State management commands require a state file. Run this command
in a directory where Terraform has been run or use the -state flag
to point the command to a specific state location.
I will be discussing this error in another issue.
I will close this issue for the time being since the main problems (using IRSA, and TerraformCloud as a backend) were resolved
What happened?
I'm trying to deploy a workspace resource from a module hosted on terraform cloud
How can we reproduce it?
Create terraform secret to be used by the ProviderConfig
Create ProviderConfig
Create the configmap that holds the values for the variables needed by the remote module
Create the workspace
What environment did it happen in?
Expected Behavior
Pull the module and deploy it as a workspace resource.
Current Behavior
getting errors like the below
Adding
https://
to the module URL returned the below error