Lazy configuration evaluation

[giovannicandido]

Hi, I have a Gradle task that creates a Postgresql container on the fly, using test containers. The URL for this container is not static; ports can change.

Then, I migrate the project using liquibase, which prepares the database for jooq code generation in another task.

The liquibase task is able to lazy load the gradle property I set after the creation of the container. But this plugin does not.

My goal is to automate the process using ci/cd. I don't want to check the generated code in version control.

Before I share part of the code, let me explain how it's done:

1. Create a new source set to add a main java application that will programmatically create the container; this sourceset should not depend on main because compileJava or compileKotlin breaks without the jooq generated code.
2. Create tasks to start and stop the postgresql container using the main code above
3. The code will create a properties file where the JDBC URL can be read
4. Create a task that will depend on startPostgresqlContainer and will read the file contents, then set a project property. After this task, the liquibase and jooq tasks wil run.
5. Configure the liquibase and jooq task to read this property by lazy evaluating it, if not set a default is readed.

The jooq task receives the default value, because the jooq configuration is evaluated at gradle configuration time, not at the task runtime.

A workaround could be to force a fixed port in the test container, but that could make the ci/cd process randomly fail.

Here is part of the code, if needed, I can share a full working example:

val TEST_ONLY_COMMON_SOURCESET = "testOnly"
val TEST_ONLY_COMMON_IMPLEMENTATION = "testOnlyImplementation"
dependencies {

//-------- TEST ONLY DEPENDENCIES
    TEST_ONLY_COMMON_IMPLEMENTATION(libs.testContainers) {
        exclude("junit:junit")
    }
    TEST_ONLY_COMMON_IMPLEMENTATION(libs.testContainersPostgres)
    TEST_ONLY_COMMON_IMPLEMENTATION(libs.testContainersJunit)
    TEST_ONLY_COMMON_IMPLEMENTATION("org.springframework.boot:spring-boot-starter-test")
    TEST_ONLY_COMMON_IMPLEMENTATION("org.springframework:spring-context")
}
sourceSets {

    create(TEST_ONLY_COMMON_SOURCESET) {

    }

    getByName("main") {
        kotlin.srcDirs.add(file("${layout.buildDirectory}/generated-src/jooq/main"))
    }
}

jooq {
    val jdbcUrl: String by lazy {
        (project.findProperty("jdbcUrl") as String?) ?: localDevDataBaseJDBCUrl
    }

    version.set(libs.versions.jooq.get())  // default (can be omitted)
    edition.set(nu.studer.gradle.jooq.JooqEdition.OSS)  // default (can be omitted)
    configurations.create("main") {
        generateSchemaSourceOnCompilation.set(false)
        jooqConfiguration.apply {
            jdbc.apply {

                driver = jdbcDriver
                url = jdbcUrl
                user = jdbcUsername
                password = jdbcPassword
            }
            generator.apply {
                name = "org.jooq.codegen.KotlinGenerator"
                database.apply {
                    name = "org.jooq.meta.postgres.PostgresDatabase"
                    inputSchema = "public"
                    recordVersionFields = ""
                    recordTimestampFields = ""
                    excludes = """
                    spatial_ref_sys
                    | databasechangelog
                    | databasechangeloglock
                """.trimIndent()
                    forcedTypes.addAll(
                        listOf(ForcedType().apply {
                            userType = "com.kugelbit.mercado.backend.domain.system_parameter.ParameterValue"
                            withJsonConverter(true)
                            includeExpression = "system_parameter\\.value"
                        })
                    )
                }
                generate.apply {
                    isDeprecated = false
                    isRecords = true
                    isImmutablePojos = true
                    isFluentSetters = true
                }
                target.apply {
                    packageName = "com.kugelbit.mercado.backend.infrastructure.database.jooqGen"
                    directory = "build/generated-src/jooq/main"

                }
                strategy.name = "org.jooq.codegen.DefaultGeneratorStrategy"
            }
        }

    }

}

tasks.named<nu.studer.gradle.jooq.JooqGenerate>("generateJooq") {
    allInputsDeclared.set(true)
    (launcher::set)(javaToolchains.launcherFor {
        languageVersion.set(JavaLanguageVersion.of(17))
    })
}

liquibase {

    val jdbcUrl: String by lazy {
        (project.findProperty("jdbcUrl") as String?) ?: localDevDataBaseJDBCUrl
    }

    activities.register("main") {

        arguments = mapOf(
            "logLevel" to "info",
            "changeLogFile" to "src/main/resources/db/changelog/db.changelog-master.xml",
            "url" to jdbcUrl,
            "username" to jdbcUsername,
            "password" to jdbcPassword,
            "driver" to jdbcDriver
        )
    }
    runList = "main"
}

tasks.register<JavaExec>("startPostgresqlContainer") {
    mainClass = "com.kugelbit.mercado.backend.test.PostgisDBContainer"
    classpath = sourceSets["testOnly"].runtimeClasspath

    val outputFile = project.layout.buildDirectory.file("testcontainer-gradle-postgis-info.properties")

    outputs.files(outputFile)
    args = arrayOf(jdbcUsername, jdbcPassword, outputFile.get().asFile.absolutePath).toMutableList()
}

tasks.register<JavaExec>("stopPostgresqlContainer") {
    mustRunAfter("update", "generateJooq")
    mainClass = "com.kugelbit.mercado.backend.test.PostgisDBContainer"
    classpath = sourceSets["testOnly"].runtimeClasspath

    val outputFile = project.layout.buildDirectory.file("testcontainer-gradle-postgis-info.properties")
    outputs.files(outputFile)
    args = arrayOf(jdbcUsername, jdbcPassword, outputFile.get().asFile.absolutePath, "stop").toMutableList()
    outputs.upToDateWhen { false }
}

tasks.register<DefaultTask>("generateJooqWithTestContainers") {
    dependsOn("startPostgresqlContainer")
    finalizedBy("update", "generateJooq")

    group = "jooq"

    doFirst {
        val postgisConfigFile = project.layout.buildDirectory.file("testcontainer-gradle-postgis-info.properties")
        val properties = Properties()
        properties.load(File(postgisConfigFile.get().asFile.absolutePath).inputStream())
        project.ext["jdbcUrl"] = properties.getProperty("jdbc")
        println("JDBC Url ${properties.get("jdbc")}")
    }

    outputs.upToDateWhen { false }

}

tasks.named("compileKotlin") {
    mustRunAfter("generateJooqWithTestContainers")
}

tasks.named("compileJava") {
    mustRunAfter("generateJooqWithTestContainers")
}

Code that creates the container:

package com.kugelbit.mercado.backend.test

import org.springframework.boot.test.util.TestPropertyValues
import org.springframework.context.ApplicationContextInitializer
import org.springframework.context.ConfigurableApplicationContext
import org.testcontainers.DockerClientFactory
import org.testcontainers.containers.PostgisContainerProvider
import org.testcontainers.junit.jupiter.Container
import java.io.File
import java.util.*
import kotlin.time.ExperimentalTime

object PostgisDBContainer {
  const val POSTGIS_TAG = "15-3.4"

  class Initialization : ApplicationContextInitializer<ConfigurableApplicationContext> {
    override fun initialize(applicationContext: ConfigurableApplicationContext) {
      val testPropertyValues = TestPropertyValues.of(
        "spring.datasource.username=" + postgis.username,
        "spring.datasource.password=" + postgis.password,
        "spring.datasource.url=" + postgis.jdbcUrl
      )
      testPropertyValues.applyTo(applicationContext)
    }

    companion object {
      @Container
      val postgis = PostgisContainerProvider(
      ).newInstance(POSTGIS_TAG)
        .withDatabaseName("mercado")
        .withUsername("root")
        .withPassword("root")
        .withReuse(false)

      init {
        postgis.start()
      }
    }
  }

  @ExperimentalTime
  @JvmStatic
  fun main(args: Array<String>) {
    val DEFAULT_USERNAME = "root"
    val DEFAULT_PASSWORD = "root"
    val DEFAULT_TEMP_OUTPUT_FILE = "/tmp/gradle-testcontainer-info.json"

    val DEFAULT_COMMAND = "start"

    var username = DEFAULT_USERNAME
    var password = DEFAULT_PASSWORD
    var filePath = DEFAULT_TEMP_OUTPUT_FILE
    var command = DEFAULT_COMMAND

    when {
      args.size == 1 -> username = args.first()
      args.size == 2 -> {
        username = args.first()
        password = args[1]
      }

      args.size == 3 -> {
        username = args.first()
        password = args[1]
        filePath = args[2]
      }

      args.size == 4 -> {
        username = args.first()
        password = args[1]
        filePath = args[2]
        command = args[3]
      }
    }

    println("Running with arguments $username, $password, $filePath, $command")

    when {
      command == "start" -> start(username, password, filePath)
      command == "stop" -> stop(filePath)
    }
  }

  fun start(username: String, password: String, filePath: String) {
    val postgis = PostgisContainerProvider(
    ).newInstance(POSTGIS_TAG)
      .withDatabaseName("mercado")
      .withUsername(username)
      .withPassword(password)
      .withReuse(true)
    postgis.start()
    println("Container ID: ${postgis.containerId}, Jdbc URL: ${postgis.jdbcUrl}")
    val file = File(filePath)
    file.createNewFile()
    val props = Properties()
    props.put("id", postgis.containerId)
    props.put("jdbc", postgis.jdbcUrl)
    props.store(file.outputStream(), null)
  }

  fun stop(configFile: String) {
    val props = Properties()
    val file = File(configFile)
    props.load(file.inputStream())

    val id = props.getProperty("id")

    val dockerClient = DockerClientFactory.instance().client();

    if (id != null) {
      println("Stopping container $id")
      dockerClient.stopContainerCmd(id).exec();
      file.delete()
    } else {
      println("Container id is not in properties file")
    }

  }
}

[etiennestuder]

Hi, what you try to achieve sounds similar to this post. I replied to it here.

[giovannicandido]

Yes, it is the same problem. The workaround works, thanks.

I set up the service using GitHub pipeline options, and that can have a static configuration.

However, if the environment does not support a fixed URL (for example, dynamic ports as the default behavior of testcontainer), then it will not work.

I tried to contribute to this change, but it will be much more work than I anticipated, so if there is more demand, this could be done.

Again, Thank you.