Howto specify backend-config to terraformInit

[farrukhnajmi]

To avoid hardcoding secrets within a S3 backend I would like to pass backend-config variables to terraformInit as described here:

https://registry.terraform.io/providers/philips-software/hsdp/latest/docs/guides/state#initialize-the-s3-backend

What is the recommended way to do that.

It seems that it is not allowed to use plan variables within the backend block so at present I do not see a way to specify the backend-config variables without hardcoding them (not a good idea as they contain secrets).

[farrukhnajmi]

Looking at code for project it seems there is no such ability. Perhaps the config could be enhnaced similar to planVariables to introduce backendConfigVariable like this:

terraform {
    ...
    sourceSets {
        main {
            ...
            backendConfigVariable("BE_VAR1", rootProject.property("BE_VAR1"))
            ...
            backendConfigVariable("BE_VARn", rootProject.property("BE_VARn"))

            planVariable("VAR1", rootProject.property("VAR1"))
            ...
            planVariable("VARn", rootProject.property("VARn"))

        }
    }
}

WDYT?

[farrukhnajmi]

@lamba92 please find patch below for addressing this issue. I followed your existing pattern for planVariables. I have locally tested this patch and it works as I proposed above and fixes my problem.

I would be most grateful if you could let me know if this fix looks good and if so, how soon is it realistic to expect this available in a release.

Index: src/main/kotlin/org/jetbrains/gradle/plugins/terraform/tasks/TerraformInit.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- src/main/kotlin/org/jetbrains/gradle/plugins/terraform/tasks/TerraformInit.kt   (revision b9add7c870c52c7cf92017ac5ba9cd4a48a32e49)
+++ src/main/kotlin/org/jetbrains/gradle/plugins/terraform/tasks/TerraformInit.kt   (date 1650034268475)
@@ -1,13 +1,11 @@
 package org.jetbrains.gradle.plugins.terraform.tasks

 import org.gradle.api.tasks.Input
+import org.gradle.api.tasks.InputFile
+import org.gradle.api.tasks.Optional
 import org.gradle.api.tasks.OutputDirectory
-import org.gradle.kotlin.dsl.getValue
-import org.gradle.kotlin.dsl.property
-import org.gradle.kotlin.dsl.provideDelegate
-import org.gradle.kotlin.dsl.setValue
-import org.jetbrains.gradle.plugins.addAll
-import org.jetbrains.gradle.plugins.property
+import org.gradle.kotlin.dsl.*
+import org.jetbrains.gradle.plugins.*
 import java.io.File

 open class TerraformInit : AbstractTerraformExec() {
@@ -15,12 +13,26 @@
     @get:Input
     var useBackend by project.objects.property(true)

+    @get:Input
+    var variables by project.objects.mapProperty<String, String?>()
+
+    @get:Input
+    var fileVariables by project.objects.mapProperty<String, File>()
+
+    @get:InputFile
+    @get:Optional
+    var variablesFile by project.objects.nullableProperty<File>()
+
     @get:OutputDirectory
     override var dataDir by project.objects.property<File>()

     override fun getTerraformArguments() = buildList<String> {
         addAll("init", "-input=false")
         if (!useBackend) add("-backend=true")
+
+        for ((k, v) in variables + fileVariables.mapValues { it.value.takeIf { it.exists() }?.readText() }) {
+            addAll("-backend-config", "$k=$v")
+        }
+        variablesFile?.run { add("-backend-config=$absolutePath") }
     }
-
 }
Index: src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformSourceSet.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformSourceSet.kt    (revision b9add7c870c52c7cf92017ac5ba9cd4a48a32e49)
+++ src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformSourceSet.kt    (date 1650032895874)
@@ -205,6 +205,84 @@
         filePlanVariables = filePlanVariables.toMutableMap().also { it[key] = value }
     }

+    /**
+     * Variables that will be used to configure backend using -backend-config to execute `terraform init`.
+     *
+     * NOTE: use [fileBackendConfigVariables] for passing variables generated from tasks.
+     */
+    var backendConfigVariables = emptyMap<String, String?>()
+
+    /**
+     * Variables that will be used to configure backend using -backend-config to execute `terraform init`.
+     * The files will be read a UTF-8 and the entire content will be used as value for the variables.
+     *
+     * Use this map to pass tasks outputs as backend-config variable for `terraform init`.
+     *
+     * Example:
+     * ```kotlin
+     *
+     * val myFile = file("output.txt")
+     * task("generator") {
+     *     doLast {
+     *         myFile.writeText("bhamamama")
+     *     }
+     * }
+     *
+     * // .....
+     *
+     * terraform {
+     *     sourceSets {
+     *         main {
+     *             fileBackendConfigVariables = mapOf("dr_kelso" to myFile)
+     *         }
+     *     }
+     * }
+     *
+     * ```
+     */
+    var fileBackendConfigVariables = emptyMap<String, File>()
+
+    /**
+     * Variable that will be read from file and used to configure backend using -backend-config  to execute `terraform init`.
+     *
+     * NOTE: use overload with [File] for passing a variable generated from a task.
+     */
+    fun backendConfigVariable(key: String, value: String?) {
+        backendConfigVariables = backendConfigVariables.toMutableMap().also { it[key] = value }
+    }
+
+    /**
+     * Variable that will be read from file and used to configure backend using -backend-config  to execute `terraform init`.
+     *
+     * NOTE: use overload with [File] for passing a variable generated from a task.
+     *
+     * Use this function to pass tasks outputs as variable for `terraform plan`.
+     *
+     * Example:
+     * ```kotlin
+     *
+     * val myFile = file("output.txt")
+     * task("generator") {
+     *     doLast {
+     *         myFile.writeText("bhamamama")
+     *     }
+     * }
+     *
+     * // .....
+     *
+     * terraform {
+     *     sourceSets {
+     *         main {
+     *             planVariable("dr_kelso", myFile)
+     *         }
+     *     }
+     * }
+     *
+     */
+    fun backendConfigVariable(key: String, value: File) {
+        fileBackendConfigVariables = fileBackendConfigVariables.toMutableMap().also { it[key] = value }
+    }
+
     /**
      * Creates a task to read an output from the TF state of the project.
      */
Index: src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformTasksContainer.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformTasksContainer.kt   (revision b9add7c870c52c7cf92017ac5ba9cd4a48a32e49)
+++ src/main/kotlin/org/jetbrains/gradle/plugins/terraform/TerraformTasksContainer.kt   (date 1650028257855)
@@ -263,6 +263,8 @@
             dependsOn(copyExecutionContext)
             attachSourceSet(sourceSet)
             finalizedBy(syncLockFile)
+            variables = sourceSet.backendConfigVariables
+            fileVariables = sourceSet.fileBackendConfigVariables
             sourceSet.tasksProvider.initActions.executeAllOn(this)
             if (terraformExtension.showInitOutputInConsole)
                 logging.captureStandardOutput(LogLevel.LIFECYCLE)

[farrukhnajmi]

@lamba92 since I have not heard back on this issue I tried another alternative that does not require any changes to plugin. The following solution works with released version 1.4.2 of plugin.

Specify only the bucket and key in the S3 backend config and specify access_key, secret_key and region via environment variables supported by terraform. Here is a sample S3 backend config:

terraform {
  backend "s3" {
    // access_key: specify via env var AWS_ACCESS_KEY_ID
    // secret_key: specify via env var AWS_SECRET_ACCESS_KEY
    // region: specify via env var AWS_REGION
    bucket = "<id of an existing s3 bucket>"
    key = "<some name to identify your app>/terraform.tfstate"
  }
}

Then set access_key, secret_key and region via environment variables in your ~/bashrc file:

export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_REGION=

Then restart the bash shell or IDE so it starts in environment with above env vars defined.

Now when terraformInit and other terraform plugin tasks are run the set the secrets correctly for S3 backend config. Reference: https://www.terraform.io/language/settings/backends/s3#credentials-and-shared-configuration

Note that it does not seem possible to specify bucket and region via env vars but that is ok as they are not sensitive secrets.