search

liquibase / liquibase-gradle-plugin

A Gradle plugin for Liquibase
Other
200 stars 59 forks source link

'classpath' parameter no longer seems to work #151

Closed rdhzl closed 1 month ago

rdhzl commented 3 months ago

We recently upgraded from 2.2.2 to 3.0.0 and after making the necessary changes to our Gradle configuration it seems that Liquibase (4.29.1) can no longer find our database changelog file.

Here's what our config looks like:

liquibase {
    activities {
        main {
            classpath "$projectDir/src/main/resources_v2/db/changelog"
            changelogFile "db.changelog-master.xml"
            url System.getenv("JDBC_DATABASE_URL") ?: testEnv.getProperty("JDBC_DATABASE_URL") ?: 'jdbc:postgresql://localhost:5432/test'
            username System.getenv("JDBC_USERNAME")
            defaultSchemaName 'hz'
        }
    }
}

For historical reasons we have our Liquibase scripts in a non-standard spot, hence the classpath.

Here's what I'm seeing

Starting Liquibase at 09:36:18 using Java 17.0.12 (version 4.29.1 #3316 built at 2024-07-30 20:15+0000)
Liquibase Version: 4.29.1
WARNING: License service not loaded, cannot determine Liquibase Pro license status. Please consider re-installing Liquibase to include all dependencies. Continuing operation without Pro license.
ERROR: Exception Details
ERROR: Exception Primary Class:  ChangeLogParseException
ERROR: Exception Primary Reason:  The file db.changelog-master.xml was not found in the configured search path:
    - /Users/rd/dev/hz-server
More locations can be added with the 'searchPath' parameter.
ERROR: Exception Primary Source:  4.29.1

Unexpected error running Liquibase: The file db.changelog-master.xml was not found in the configured search path:
    - /Users/rd/dev/hz-server

Things work as expected if I bump back down to 2.2.2.

Nothing jumped out at me in the release notes around this. Am I missing something?

Thanks for your assistance.

stevesaliman commented 3 months ago

The Liquibase Gradle plugin is designed to be a pass-through to Liquibase itself for most things. If you run with --debug, does the log file show that the plugin is properly passing the classpath to Liquibase itself?

Other things that come to mind:

  1. What happens if you change the changelogFile to "classpath:db.changelog-master.xml"?
  2. What happens if you add "$projectDir/src/main/resources_v2/db/changelog" to the liquibaseRuntime configuration?
rdhzl commented 3 months ago

Hi Steve,

Thanks for the response.

I do notice the following in the debug log output in v3.0.0 that does not appear with 2.2.2:

3.0.0

2024-08-06T09:19:00.432-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute exec for :recreateTestDB' completed
2024-08-06T09:19:00.433-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
2024-08-06T09:19:00.433-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Realize task :update' completed
2024-08-06T09:19:00.435-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.AutoloadedConfigurations instance liquibase.GlobalConfiguration
2024-08-06T09:19:00.435-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.AutoloadedConfigurations instance liquibase.integration.commandline.LiquibaseCommandLineConfiguration
2024-08-06T09:19:00.435-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.AutoloadedConfigurations instance liquibase.logging.core.DefaultLoggerConfiguration
2024-08-06T09:19:00.435-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.AutoloadedConfigurations instance liquibase.sql.SqlConfiguration
2024-08-06T09:19:00.435-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.AutoloadedConfigurations instance liquibase.parser.ChangeLogParserConfiguration
2024-08-06T09:19:00.436-0400 [DEBUG] [liquibase.configuration] Found ConfigurationDefinitions in liquibase.GlobalConfiguration
2024-08-06T09:19:00.436-0400 [DEBUG] [liquibase.configuration] Found ConfigurationDefinitions in liquibase.integration.commandline.LiquibaseCommandLineConfiguration
2024-08-06T09:19:00.436-0400 [DEBUG] [liquibase.configuration] Found ConfigurationDefinitions in liquibase.logging.core.DefaultLoggerConfiguration
2024-08-06T09:19:00.437-0400 [DEBUG] [liquibase.configuration] Found ConfigurationDefinitions in liquibase.sql.SqlConfiguration
2024-08-06T09:19:00.437-0400 [DEBUG] [liquibase.configuration] Found ConfigurationDefinitions in liquibase.parser.ChangeLogParserConfiguration
2024-08-06T09:19:00.437-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.ConfigurationValueProvider instance liquibase.configuration.core.DeprecatedConfigurationValueProvider
2024-08-06T09:19:00.437-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.ConfigurationValueProvider instance liquibase.configuration.core.SystemPropertyValueProvider
2024-08-06T09:19:00.438-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.ConfigurationValueProvider instance liquibase.configuration.core.ScopeValueProvider
2024-08-06T09:19:00.438-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.configuration.ConfigurationValueProvider instance liquibase.configuration.core.EnvironmentValueProvider
2024-08-06T09:19:00.439-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.logging.LogService instance liquibase.logging.core.BufferedLogService
2024-08-06T09:19:00.439-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.logging.LogService instance liquibase.logging.core.CompositeLogService
2024-08-06T09:19:00.439-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.logging.LogService instance liquibase.logging.core.JavaLogService
2024-08-06T09:19:00.440-0400 [DEBUG] [liquibase.servicelocator] Loaded liquibase.servicelocator.ServiceLocator instance liquibase.servicelocator.StandardServiceLocator
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the changelogFile command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the classpath command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the defaultSchemaName command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the logLevel command argument because it is not supported by the update
2024-08-06T09:19:00.454-0400 [DEBUG] [org.gradle.api.Project] skipping the url command argument because it is not supported by the update
2024-08-06T09:19:00.454-0400 [DEBUG] [org.gradle.api.Project] skipping the username command argument because it is not supported by the update
2024-08-06T09:19:00.456-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.ResolvedArtifactCaches] Reusing in-memory cache for repo 'MavenRepo' [26c913274550a0b2221f47a0fe2d2358].
2024-08-06T09:19:00.457-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.builder.DependencyGraphBuilder] Visiting configuration :hippo:unspecified(liquibaseRuntime).
2024-08-06T09:19:00.457-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for org.liquibase:liquibase-core:4.29.1 using repositories [MavenRepo]

2.2.2

2024-08-06T09:29:07.069-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute exec for :recreateTestDB' completed
2024-08-06T09:29:07.070-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Execute container callback action' completed
2024-08-06T09:29:07.070-0400 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Realize task :update' completed
2024-08-06T09:29:07.073-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.ResolvedArtifactCaches] Reusing in-memory cache for repo 'MavenRepo' [26c913274550a0b2221f47a0fe2d2358].
2024-08-06T09:29:07.074-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.builder.DependencyGraphBuilder] Visiting configuration :hippo:unspecified(liquibaseRuntime).
2024-08-06T09:29:07.074-0400 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for org.liquibase:liquibase-core:4.29.1 using repositories [MavenRepo]

I tried prefixing the classpath as you suggested and similar results. I don't currently have a liquibaseRuntime block so I wasn't exactly certain how to address your second point.

Let me know if there's additional detail you need from the logs. Thanks again for your assistance.

stevesaliman commented 3 months ago

The line I was hoping to see starts with liquibase-plugin: Running 'liquibase, and it tells us exactly what is being sent to Liquibase itself. If that line looks good, then we need to see why Liquibase isn't finding things. If that line is not correct, then we need to figure out why the plugin isn't calling Liquibase correctly.

I'm curious about the "I don't currently have a liquibaseRuntime block" comment... When the plugin runs Liquibase itself, it uses the liquibaseRuntime configuration to find things like the Groovy DSL, database drivers, and Liquibase itself. That part hasn't changed from 2.2.2, so I'm curious to know how it ran at all in the past.

I also notice you're running Liquibase 4.29... Testing was done in 4.26. Do you get a different result with 4.26?

rdhzl commented 3 months ago

2.2.2 + LB 2.29.1 2024-08-06T09:29:07.082-0400 [DEBUG] [org.gradle.api.Project] liquibase-plugin: Running 'liquibase --log-level=info --classpath=/Users/rd/dev/hz-server/src/main/resources_v2/db/changelog --changelog-file=db.changelog-master.xml --url=jdbc:postgresql://localhost:5432/test --username=postgres --default-schema-name=hz update'

3.0.0 + LB 2.29.1 2024-08-06T09:19:00.460-0400 [DEBUG] [org.gradle.api.Project] liquibase-plugin: Running 'liquibase update'

3.0.0 + LB 2.26.0 2024-08-06T10:44:21.868-0400 [DEBUG] [org.gradle.api.Project] liquibase-plugin: Running 'liquibase --classpath=/Users/rd/dev/hz-server/src/main/resources_v2/db/changelog --log-level=info update --changelog-file=db.changelog-master.xml --default-schema-name=hz --url=jdbc:postgresql://localhost:5432/test --username=postgres'

So it does seem like there's a difference of behavior between 2.26/2.29.

I probably misspoke around the liquibaseRuntime block. I have the following:

    liquibaseRuntime libs.liquibaseCore
    liquibaseRuntime "org.postgresql:postgresql:$postgresqlDriver"
    liquibaseRuntime 'info.picocli:picocli:4.7.6'
stevesaliman commented 3 months ago

At first glance, it would appear that Liquibase has changed the Command API the plugin is using to get the supported arguments of a command. I'll try to dig in more when I get some time.

stevesaliman commented 3 months ago

At second glance, I think something else is going on...

I have a project that uses Liquibase 4.29, and it calls Liquibase correctly. So the next step is to figure out what the difference could be between our two projects.

Is it possible that buildscript classpath and the liquibaseRuntime classpath are using the same version of Liquibase?

rdhzl commented 3 months ago

Hi Steve,

Here are, I think, the relevant bits:

settings.gradle

dependencyResolutionManagement {
    versionCatalogs {
        libs {
            library('liquibaseCore', 'org.liquibase:liquibase-core:4.29.1')
        }
    }
}

build.gradle

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath libs.liquibaseCore
    }
}

plugins {
    id 'org.liquibase.gradle' version '3.0.0'
}
dependencies {
    liquibaseRuntime libs.liquibaseCore
    liquibaseRuntime "org.postgresql:postgresql:$postgresqlDriver"
    liquibaseRuntime 'info.picocli:picocli:4.7.6'
    implementation libs.liquibaseCore
}
liquibase {
    activities {
        main {
            classpath "$projectDir/src/main/resources_v2/db/changelog"
            changelogFile "db.changelog-master.xml"
            url System.getenv("JDBC_DATABASE_URL")
            username 'postgres'
            defaultSchemaName 'hz'
        }
    }
}

We didn't have a buildScript block prior to this but added based on the new requirements. So I also took the opportunity to add the shared dependency version to settings.gradle (Previously was in an ext block).

If I bump the version in settings.gradle back down to 4.26.0 things work. The dependency graph only shows the expected version. Gradle behavior is the same when run via IntelliJ or the command line. We use Gradle 8.7 at present. I've run gradle clean after each dependency swap.

I will say that the Gradle task I originally noticed this issue was a custom one where we call a shell script to drop/recreate a Postgress schema and then call Liquibase plug-in's update (via update.exec()) to bring it up to date.

That being said if I just call the Liquibase plug-in's update target directly I get the same behavior. It works with 4.26.0 and 4.29.1 complains about not finding the db changelog. It seems like the contents of my liquibase config block are being ignored by 3.0.0, based on the log messages from above...

2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the changelogFile command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the classpath command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the defaultSchemaName command argument because it is not supported by the update
2024-08-06T09:19:00.452-0400 [DEBUG] [org.gradle.api.Project] skipping the logLevel command argument because it is not supported by the update
2024-08-06T09:19:00.454-0400 [DEBUG] [org.gradle.api.Project] skipping the url command argument because it is not supported by the update
2024-08-06T09:19:00.454-0400 [DEBUG] [org.gradle.api.Project] skipping the username command argument because it is not supported by the update

Is there something about my liquibase block that's perhaps now incompatible?

Thanks for looking into this. It's not really a criticality on our end at the moment, can still work with 2.2.2. If there's additional information I can provide let me know. I'll admit to not being a Gradle maven, I know enough to get by so it may be that I'm doing something wrong.

stevesaliman commented 3 months ago

I don't see anything obviously wrong with your liquibase block...

The way that the plugin now works is that when you chose to run a task, like update, the plugin asks Liquibase what the supported arguments of the command are, as well as what the "global" arguments are so that it knows what the supported arguments are.

It then looks at each property in the activity (classpath, url, etc.), to see which list it is in so that the plugin knows if it needs to be sent to Liquibase before or after the command. If there is anything in your liquibase block that wasn't in either of the "supported" lists it got back from Liquibase, the plugin ignores it, and you get the "Skipping" message you are seeing.

The behavior you are seeing suggests that Liquibase is returning empty lists for both of the requests, but only in Liquibase 4.29.1 (does it work in 4.29.0?). What I can't figure out is why it works in my test project. What subtle thing in my class path is different from your class path?

rdhzl commented 3 months ago

I tried bumping to a few different versions of LB. It seems like anything after 4.26.0 fails for me when coupled with 3.0.0 of the plug-in.

Just to confirm, the config nesting of liquibase.activities.main is still correct from your perspective?

It could be something innocuous in my Gradle build or maybe someone with better context will run into this and add commentary.

stevesaliman commented 3 months ago

The only thing that changed in liquibase.activities.main from version 2.x to 3.x of the plugin is that changeLogFile became changelogFile (the case of the L changed), which you've fixed correctly. Everything else looks good there.

I'm also hoping someone else with better context can shed some light on this. My hunch is that there is some subtle classpath difference between your project and mine that causes issues, but only in LB 4.27+

mlanglet commented 2 months ago

We're seeing the same problem with classpath

ERROR: Exception Details ERROR: Exception Primary Class: ChangeLogParseException ERROR: Exception Primary Reason: resources/db/changelog/changelog-root.yml does not exist ERROR: Exception Primary Source: 4.29.1

Unexpected error running Liquibase: resources/db/changelog/changelog-root.yml does not exist

All from build.gradle of the gradle module that needs to use the plugin during build:

plugins {
  // ...
  id("org.liquibase.gradle") version "3.0.0"
}
buildscript {
   repositories {
       mavenCentral()
   }
   dependencies {
       classpath(libs.liquibase.core)
   }
}
dependencies {
  // ...
  liquibaseRuntime(libs.liquibase.core)
  liquibaseRuntime(libs.postgres)
  liquibaseRuntime(libs.picocli)
}
liquibase {
   activities.register("compose") {
       this.arguments =
           mapOf(
               "classpath" to "${rootProject.rootDir}/lib/db/src/main",
               "changelogFile" to "resources/db/changelog/changelog-root.yml",
               // ...
           )
   }
}

Same config works for 2.2.2

Any news on this?

frjonsen commented 2 months ago

We seem to be running into the same issue, with everything configured the same as @mlanglet and receiving the same error.

Already spent quite a lot of time trying to resolve this without really getting anywhere, so I guess we'll be staying on 2.X for now.

rossdanderson commented 2 months ago

Also seeing all of our arguments ignored attempting to upgrade with 4.27.0 and plugin 3.0.0

2024-08-27T12:09:57.390+0100 [DEBUG] [org.gradle.api.Project] skipping the changelogFile command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the contexts command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the labels command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the log-level command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the password command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the url command argument because it is not supported by the dropAll
2024-08-27T12:09:57.391+0100 [DEBUG] [org.gradle.api.Project] skipping the username command argument because it is not supported by the dropAll
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.liquibase:liquibase-core:4.27.0")
    }
}
dependencies {
    api(project(":common"))

    jooqGenerator("org.mariadb.jdbc:mariadb-java-client")

    liquibaseRuntime("org.liquibase:liquibase-core:4.27.0") // Using Spring 3.3.3
    liquibaseRuntime("org.mariadb.jdbc:mariadb-java-client")
    liquibaseRuntime("ch.qos.logback:logback-core")
    liquibaseRuntime("ch.qos.logback:logback-classic")
    liquibaseRuntime("info.picocli:picocli:4.6.1") // Also tried the latest
    liquibaseRuntime(project(":common"))

    api("org.jooq:jooq")
}

Attempting to use Kotlin DSL

liquibase {
    val main = activities.create("main")
    main.arguments = mapOf(
        "url" to "jdbc:mariadb://$dbHost:$dbPort/nomad?connectionTimeZone=LOCAL&forceConnectionTimeZoneToSession=true",
        "contexts" to (project.findProperty("liquibase.contexts") ?: "dev").toString(),
        "labels" to (project.findProperty("liquibase.labels") ?: "trep or trbs or 360t").toString(),
        "changelogFile" to "engine/src/main/resources/db/changelog/changelog-master.xml",
        "username" to dbUser,
        "password" to dbPassword,
        "log-level" to "warn",
    )
}
stevesaliman commented 2 months ago

Before the plugin runs the first Liquibase related task, it asks Liquibase for the arguments it supports. When the plugin logs that it is ignoring an argument, it is because liquibase has not returned it as a valid argument. The question is why?

The plugin gets command arguments with the getArguments() method of a CommandDefinition. It gets global arguments with:

SortedSet<ConfigurationDefinition<?>> globalConfigurations = Scope
        .getCurrentScope()
        .getSingleton(LiquibaseConfiguration.class)
        .getRegisteredDefinitions(false);

This is also how the Liquibase CLI gets them. The question is, why does this return an empty list for some people and not others?

stevesaliman commented 2 months ago

This is getting a whole lot weirder. Thanks to @marques-work, who reported what I think is the same issue in #154, I can now reproduce the problem on my end.

However, the problem doesn't happen when I run with the debugger. I have no idea why I'd get different results with and without it, but here we are.

I also see two problems to fix. The first is that the global arguments are coming back empty, which is why things like searchPath get ignored, and the second is that when running outside the debugger, the CommandDefinition.getArguments() call is not returning anything either.

I'll keep looking into this, but if anyone has any ideas, I'd love to hear them.

stevesaliman commented 1 month ago

I may have figured out what is going on. I pushed an update to the code, but I haven't released a new version yet - I want to see of it fixes the problem. @rossdanderson @frjonsen @mlanglet if you build a local copy of the plugin, does it fix your issue?

mlanglet commented 1 month ago

I tried to clone the repo, run the publishToMavenLocal, updated my buildscript with mavenLocal and updated my plugin version to 3.0.1-SNAPSHOT. I can see the snapshot build in my .m2 directory but I still get this error:

Plugin Repositories (could not resolve plugin artifact 'org.liquibase.gradle:org.liquibase.gradle.gradle.plugin:3.0.1-SNAPSHOT')
  Searched in the following repositories:
    MavenLocal(file:/Users/mathiaslanglet/.m2/repository/)
    Gradle Central Plugin Repository
buildscript {
    repositories {
        mavenLocal()
        mavenCentral()
    }
    dependencies {
        classpath(libs.liquibase.core)
    }
}

plugins {
    // ...
    id("org.liquibase.gradle") version "3.0.1-SNAPSHOT"
}

Any idea what I might be doing wrong?

stevesaliman commented 1 month ago

Does your project also have a settings.gradle or settings.gradle.kts file? If so, you may need to have the following in it:

pluginManagement {
    repositories {
        mavenLocal()
        gradlePluginPortal()
    }
}
mlanglet commented 1 month ago

Thanks, that didn't do it for me though. Strange.. I'm so confused right now :D

stevesaliman commented 1 month ago

Others have reported that the new release calls Liquibase correctly and fixes the bug. I've released version 3.0.1 of the plugin to the Gradle plugin portal, it should now work for all.

rdhzl commented 1 month ago

Thanks for your efforts, @stevesaliman!

mlanglet commented 1 month ago

Great stuff, thanks @stevesaliman 🙌