Make the backend configurable

[chanwit]

Currently we support only Kubernetes backend. But some users still require to use their existing backends like S3 or Dynamo tables outside Kubernetes.

It would be great to make the backend configurable to support those existing systems.

Here's a possible API design for custom backend.

---
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: tf-controller-ng-cc5a1ac4
  namespace: flux-system
spec:
  interval: 1h
  path: ./_artifacts/20-zz-terraform
  approvePlan: "auto"
  backend:
    customConfiguration: |
      backend "s3" {
        bucket = "mybucket"
        key    = "path/to/my/key"
        region = "us-east-1"
      }
  varsFrom:
  - kind: Secret
    name: aws-vars
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system

[fsequeira1]

Hi, I am interested in working on this

[chanwit]

Thank you @fsequeira1

[chanwit]

@fsequeira1 have you got a chance to tackle this issue please?

[fsequeira1]

@chanwit it is not forgotten but still haven't got an opportunity to work on it

[chanwit]

Thank you, @fsequeira1. I just wanted to check about it, as it's also in the roadmap of Q1, which is 15 days left.

But I can postpone it to Q2 if you think it won't be landing by the Q1 milestone. Please feel free to let me know which one works for you.

[fsequeira1]

@chanwit I think it's better to postpone to Q2 but I will try to deliver it asap

[chanwit]

Thank you @fsequeira1 I'll move it to Q2.

[chanwit]

@fsequeira1 here's a design of the customDefinition block that could help you speeding things up. Please define customDefinition as *string and if it's specified, you can dump the whole string to override the backend block, for example.

---
apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
  name: tf-controller-ng-cc5a1ac4
  namespace: flux-system
spec:
  backend:
    customConfiguration: |
      backend "s3" {
        bucket = "mybucket"
        key    = "path/to/my/key"
        region = "us-east-1"
      }

[o6uoq]

Assuming this implies Terraform Remote Backends, as listed here ?

If so, +1 for S3 and Consul as they are the most common backends in my experience with customers.

[chanwit]

Yes it is

[surskitt]

Is this still being worked on? We'd much rather continue to use our existing state than have the state within the kubernetes cluster (especially considering the kubernetes infra itself is managed using terraform).

[chanwit]

@fsequeira1 have you got any update on this please?

[fsequeira1]

@chanwit I can create a pr with the backend part, but it still lacks validation of the backend and support for secrets.

[chanwit]

fixed by #288