Simplified backend management

[frank-bee]

Currently the state is either stored in a secret, which is cumbersome to backup / version. Or it is stored as an external backend ( e.g. on s3) which is cumbersome to setup ( a lot of code duplication , many state files potentially ).

Proposal: Allow configuring the backend on controller level, everything besides the "key", e.g. :

  backendConfig:
    customConfiguration: |
      backend "s3" {
        bucket                      = "s3-terraform-state1"
        region                      = "us-east-1"
        endpoint                    = "http://localhost:4566"
        skip_credentials_validation = true
        skip_metadata_api_check     = true
        force_path_style            = true
        dynamodb_endpoint           = "http://localhost:4566"
        encrypt                     = true
      }
...
  }

And create the actual state files ( "key"s ) automatically. (Like tools like terragrunt allow this, If I remember that correctly)

Even the state lock table ( in AWS it is a dynamo table ) could be created automatically.

=========

User story:

As a Terraform Controller developer, I'd like to simplify the backend management for Terraform Controller, so that users can configure the backend on the controller level and have state files managed more easily.

Acceptance Criteria:

• [ ] Users can specify a backend configuration at the controller level as described above.
• [ ] State files ("key"s) are created and managed automatically by the controller without user intervention.
• [ ] State lock tables (e.g. DynamoDB tables for AWS) are created and managed automatically by the controller.

[frank-bee]

See here, how terragrunt handles this feature https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/#create-remote-state-and-locking-resources-automatically

[chanwit]

Thank you @frank-bee This seems to be a great feature that improves UX of the controller a lot.

[LappleApple]

Closing for now as it's out of scope for our immediate plans. Will reopen should this change.