terraform: `Workspace` CRD is not working with an http backend in gitlab

ecthelion77 commented 1 year ago

What happened?

When i try to use an http backend for state on gitlab, I have the following error message in my workspace object :

cannot select Terraform workspace: failed to get configured named states: workspaces not supported

Configuration is working well when

I execute locally on my laptop
in a gitlab pipeline
manually run with the docker image used by the TF provider :

How can we reproduce it?

Provider config

apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: >-
      dev-204-cp-az-pim-mgmt:tf.upbound.io/ProviderConfig:cp-az-pim-mgmt/cp-az-pim-mgmt
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"tf.upbound.io/v1beta1","kind":"ProviderConfig","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"dev-204-cp-az-pim-mgmt:tf.upbound.io/ProviderConfig:cp-az-pim-mgmt/cp-az-pim-mgmt"},"labels":{"app.kubernetes.io/instance":"dev-204-cp-az-pim-mgmt","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"cp-az-pim-mgmt","helm.sh/chart":"addons-1.0.0"},"name":"cp-az-pim-mgmt"},"spec":{"configuration":null,"credentials":[{"filename":".git-credentials","secretRef":{"key":".git-credentials","name":"dev-204-cp-az-pim-mgmt-gitlab-pat","namespace":"cp-az-pim-mgmt"},"source":"Secret"},{"filename":"backend.tfvars","secretRef":{"key":"backend.tfvars","name":"dev-204-cp-az-pim-mgmt-gitlab-pat","namespace":"cp-az-pim-mgmt"},"source":"Secret"},{"filename":"azure-provider-config.tf.json","secretRef":{"key":"azure-provider-config.tf.json","name":"dev-204-cp-az-pim-mgmt-az-provider-config","namespace":"cp-az-pim-mgmt"},"source":"Secret"}]}}
  creationTimestamp: '2022-12-19T15:46:40Z'
  finalizers:
    - in-use.crossplane.io
  generation: 5
  labels:
  name: cp-az-pim-mgmt
spec:
  credentials:
    - filename: .git-credentials
      secretRef:
        key: .git-credentials
        name: dev-204-cp-az-pim-mgmt-gitlab-pat
        namespace: cp-az-pim-mgmt
      source: Secret
    - filename: backend.tfvars
      secretRef:
        key: backend.tfvars
        name: dev-204-cp-az-pim-mgmt-gitlab-pat
        namespace: cp-az-pim-mgmt
      source: Secret
    - filename: azure-provider-config.tf.json
      secretRef:
        key: azure-provider-config.tf.json
        name: dev-204-cp-az-pim-mgmt-az-provider-config
        namespace: cp-az-pim-mgmt
      source: Secret
  pluginCache: true
status:
  users: 1

Workspace config :

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: >-
      dev-204-cp-az-pim-mgmt:tf.upbound.io/Workspace:cp-az-pim-mgmt/cp-az-pim-mgmt
    crossplane.io/external-name: aws-clz
  labels:
    app.kubernetes.io/instance: dev-204-cp-az-pim-mgmt
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cp-az-pim-mgmt
    helm.sh/chart: addons-1.0.0
  name: cp-az-pim-mgmt
spec:
  deletionPolicy: Delete
  forProvider:
    entrypoint: ''
    initArgs:
      - '-backend-config=backend.tfvars'
    module: 'git::https://gitlab.com/xxxxxx/xxxxxxx/xxxxx?ref=master'
    source: Remote
    vars:
      - key: TF_STATE_NAME
        value: default
  providerConfigRef:
    name: cp-az-pim-mgmt
  writeConnectionSecretToRef:
    name: cp-az-pim-mgmt
    namespace: cp-az-pim-mgmt
status:
  atProvider: {}
  conditions:
    - lastTransitionTime: '2022-12-20T10:59:14Z'
      message: >-
        connect failed: cannot select Terraform workspace: failed to get
        configured named states: workspaces not supported
      reason: ReconcileError
      status: 'False'
      type: Synced

backend.tfvars :

address         = "https://gitlab.com/api/v4/projects/xxxxxxxxxxxxxx/terraform/state/default"
lock_address    = "https://gitlab.com/api/v4/projects/xxxxxxxxxxxxxx/terraform/state/default/lock"
unlock_address  = "https://gitlab.com/api/v4/projects/xxxxxxxxxxxxxx/terraform/state/default/lock"
username        = "xxxxxxxxxxxxxx"
password        = "xxxxxxxxxxxxxx"
lock_method     = "POST"
unlock_method   = "DELETE"
retry_wait_min  = 5

What environment did it happen in?

Universal Crossplane Version: upbound/crossplane:v1.10.1-up.1
Provider Version: xpkg.upbound.io/upbound/provider-terraform:v0.2.0
Cloud provider or hardware configuration : on-premise
Kubernetes version : 1.22.7
OS : ubuntu 20.04

bobh66 commented 1 year ago

This is interesting - the http backend does not support workspaces, it only allows the workspace name "default". The provider uses workspaces for separation of user data, one workspace per Workspace object, with the UUID as the name of the workspace, and it selects the current workspace name before doing any work on it.

Somehow the provider would need to "know" that the Workspace is using an HTTP backend and not try to select the workspace by name. I don't think we have done much with remote backends yet (obviously). @negz @ytsarev - any thoughts?

ytsarev commented 1 year ago

@bobh66 I tested provider-terraform with remote s3 backend back in Absa, and it worked out of the box...

I didn't dig much into this specific case but maybe we can set the workspace name dynamically via https://developer.hashicorp.com/terraform/language/settings/backends/http#address ?

bobh66 commented 1 year ago

@ytsarev I think the problem is caused by the "terraform workspace select" command that the provider uses to set the workspace to the UUID value. That command seems to fail when the backend is http because only the default workspace is allowed for that backend. If we allowed the user to somehow indicate that terraform workspace select should not be called, we risk having multiple Workspaces using the same "default" backend if it is configured improperly. Or we ignore the error for that specific case, but that doesn't seem like a good idea either.

Other than not supporting the http backend I don't have any good workarounds - ideas?

upbound / provider-terraform