Open sdavids opened 4 months ago
When you're using the Docker Compose support, the auto-configured DataSource
is created using the service connection details that comes from any SQL database service. These details are the JDBC URL, the username, and the password. They intentionally take precedence over the JDBC URL, username, and password configured in application.yaml
as those details won't, typically, allow a connection to the Docker Compose-managed database to be established.
Can you please describe what you're trying to achieve here? I think I might be able to reverse engineer it from the scripts and configuration that you have shared, but a description directly from you will be considerably more accurate.
The setup above in words:
Postgres instance; admin super user 'sa'
Each bounded context gets its own database and super user; database 'example`, super user 'example_admin'.
Each bounded context creates one or more schemas in its database and uses a dedicated admin user for each schema—all created database objects will be owned by the admin user; schema 'example`, admin user 'example_ow'
Two additional users: One with read-only permissions and one with read-write (but not create) permissions; user 'example_ro' and user 'example_rw'.
Each bounded context will be provisioned by Liquibase with the corresponding admin user; admin user 'example_ow'.
The bounded context's application uses the read-write user; user 'example_rw'.
Other stuff (Reporting/QA/etc.) uses the read-only user; user 'example_ro'.
The basic premise is: The development setup should be as close as possible to production.
As it is now, everything is done with the super user 'sa'—therefore no permission checks are performed because 'sa' has all permissions.
Therefore, if you forget to setup permissions correctly in the Liquibase migration scripts the local development setup will work.
Once you deploy to production it might not work because you set the permissions incorrectly (or forgot to set them up altogether), i.e. missing GRANT
statements.
TL;DR
By using the super user one cannot test if the database roles and permissions are set up correctly.
At first, I tried replicating our setup with Using Testcontainers at Development Time but unfortunately the Testcontainers Postgres support does not allow multiple init scripts running against different databases:
https://github.com/testcontainers/testcontainers-java/issues/8634
Thanks for the additional details. While cumbersome, I think you can achieve what you want with a custom connection details factory:
package com.example;
import org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails;
import org.springframework.boot.docker.compose.core.RunningService;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionDetailsFactory;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionSource;
import org.springframework.boot.docker.compose.service.connection.jdbc.JdbcUrlBuilder;
import org.springframework.core.Ordered;
class CustomUsernameAndPasswordPostgresJdbcDockerComposeConnectionDetailsFactory
extends DockerComposeConnectionDetailsFactory<JdbcConnectionDetails> implements Ordered {
protected CustomUsernameAndPasswordPostgresJdbcDockerComposeConnectionDetailsFactory() {
super("postgres");
}
@Override
protected JdbcConnectionDetails getDockerComposeConnectionDetails(DockerComposeConnectionSource source) {
return new PostgresJdbcDockerComposeConnectionDetails(source.getRunningService());
}
static class PostgresJdbcDockerComposeConnectionDetails extends DockerComposeConnectionDetails
implements JdbcConnectionDetails {
private static final JdbcUrlBuilder jdbcUrlBuilder = new JdbcUrlBuilder("postgresql", 5432);
private final String jdbcUrl;
PostgresJdbcDockerComposeConnectionDetails(RunningService service) {
super(service);
this.jdbcUrl = jdbcUrlBuilder.build(service, "example");
}
@Override
public String getUsername() {
return "example_rw";
}
@Override
public String getPassword() {
return "example_rw";
}
@Override
public String getJdbcUrl() {
return this.jdbcUrl;
}
}
@Override
public int getOrder() {
return 0;
}
}
Registered in META-INF/spring.factories
under the org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory
key, this should give you complete control of mapping from the running Docker Compose service to the JDBC connection details. You could do similar for Liquibase by implementing another factory that produces LiquibaseConnectionDetails
.
Note that ConnectionDetailsFactory
implementations don't have access to the Environment
so you'd have to either hardcode the username and password or provide them via some other means for now at least.
I will try it later today, thanks …
I tried your suggestion above:
java.lang.IllegalStateException: Duplicate connection details supplied for org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails
at org.springframework.util.Assert.state(Assert.java:97) ~[spring-core-6.1.6.jar:6.1.6]
at org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactories.getConnectionDetails(ConnectionDetailsFactories.java:84) ~[spring-boot-autoconfigure-3.2.5.jar:3.2.5]
at org.springframework.boot.docker.compose.service.connection.DockerComposeServiceConnectionsApplicationListener.registerConnectionDetails(DockerComposeServiceConnectionsApplicationListener.java:68) ~[spring-boot-docker-compose-3.2.5.jar:3.2.5]
package com.example;
import org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails;
import org.springframework.boot.docker.compose.core.RunningService;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionDetailsFactory;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionSource;
import org.springframework.boot.docker.compose.service.connection.jdbc.JdbcUrlBuilder;
import org.springframework.core.Ordered;
class CustomPostgresJdbcDockerComposeConnectionDetailsFactory
extends DockerComposeConnectionDetailsFactory<JdbcConnectionDetails> implements Ordered {
CustomPostgresJdbcDockerComposeConnectionDetailsFactory() {
super("postgres");
}
@Override
public int getOrder() {
return 0;
}
@Override
protected JdbcConnectionDetails getDockerComposeConnectionDetails(
DockerComposeConnectionSource source) {
return new PostgresJdbcDockerComposeConnectionDetails(source.getRunningService());
}
static class PostgresJdbcDockerComposeConnectionDetails
extends DockerComposeConnectionDetailsFactory.DockerComposeConnectionDetails
implements JdbcConnectionDetails {
private final String jdbcUrl;
PostgresJdbcDockerComposeConnectionDetails(RunningService service) {
super(service);
this.jdbcUrl = new JdbcUrlBuilder("postgresql", 5432).build(service, "example");
}
@Override
public String getUsername() {
return "example_rw";
}
@Override
public String getPassword() {
return "example_rw";
}
@Override
public String getJdbcUrl() {
return this.jdbcUrl;
}
}
}
org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory=\
com.example.CustomPostgresJdbcDockerComposeConnectionDetailsFactory
The current registration algorithm does not support Ordered
.
Sorry, I'd incorrectly recalled that the first factory would win, hence it implementing Ordered
.
You can get PostgresJdbcDockerComposeConnectionDetailsFactory
to ignore your db
service by hiding the fact that it's Postgres. This will then allow the custom factory to take control. First, add a org.springframework.boot.service-connection
label to the service with a value that's anything other than postgres
, say custom-postgres
. Then update the constructor of CustomPostgresJdbcDockerComposeConnectionDetailsFactory
to pass that value to its super constructor:
protected CustomUsernameAndPasswordPostgresJdbcDockerComposeConnectionDetailsFactory() {
super("custom-postgres");
}
This should ensure that only the custom factory is used for your db
service.
package com.example;
import org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails;
import org.springframework.boot.docker.compose.core.RunningService;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionDetailsFactory;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionSource;
import org.springframework.boot.docker.compose.service.connection.jdbc.JdbcUrlBuilder;
class CustomDockerComposeConnectionDetailsFactory
extends DockerComposeConnectionDetailsFactory<JdbcConnectionDetails> {
CustomDockerComposeConnectionDetailsFactory() {
super("custom-postgres");
}
@Override
protected JdbcConnectionDetails getDockerComposeConnectionDetails(
DockerComposeConnectionSource source) {
return new CustomDockerComposeConnectionDetails(source.getRunningService());
}
static class CustomDockerComposeConnectionDetails extends DockerComposeConnectionDetails
implements JdbcConnectionDetails {
private final String jdbcUrl;
CustomDockerComposeConnectionDetails(RunningService service) {
super(service);
this.jdbcUrl = new JdbcUrlBuilder("postgresql", 5432).build(service, "example");
}
@Override
public String getUsername() {
return "example_rw";
}
@Override
public String getPassword() {
return "example_rw";
}
@Override
public String getJdbcUrl() {
return jdbcUrl;
}
}
}
org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory=\
com.example.CustomDockerComposeConnectionDetailsFactory
services:
db:
...
labels:
org.springframework.boot.service-connection: 'custom-postgres'
The setup above works.
But once Liquibase is in the mix we are back at square one.
package com.example;
import org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails;
import org.springframework.boot.autoconfigure.liquibase.LiquibaseConnectionDetails;
import org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory;
class CustomLiquibaseConnectionDetailsFactory
implements ConnectionDetailsFactory<JdbcConnectionDetails, LiquibaseConnectionDetails> {
CustomLiquibaseConnectionDetailsFactory() {}
@Override
public LiquibaseConnectionDetails getConnectionDetails(JdbcConnectionDetails input) {
return new MyLiquibaseConnectionDetails(input);
}
static class MyLiquibaseConnectionDetails implements LiquibaseConnectionDetails {
private final String jdbcUrl;
private final String driverClassName;
public MyLiquibaseConnectionDetails(JdbcConnectionDetails input) {
jdbcUrl = input.getJdbcUrl();
driverClassName = input.getDriverClassName();
}
@Override
public String getUsername() {
return "example_ow";
}
@Override
public String getPassword() {
return "example_ow";
}
@Override
public String getJdbcUrl() {
return jdbcUrl;
}
@Override
public String getDriverClassName() {
return driverClassName;
}
}
}
org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory=\
com.example.CustomDockerComposeConnectionDetailsFactory,\
com.example.CustomLiquibaseConnectionDetailsFactory
⇓
java.lang.IllegalStateException: Duplicate connection details supplied for org.springframework.boot.autoconfigure.liquibase.LiquibaseConnectionDetails
at org.springframework.util.Assert.state(Assert.java:97) ~[spring-core-6.1.6.jar:6.1.6]
at org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactories.getConnectionDetails(ConnectionDetailsFactories.java:84) ~[spring-boot-autoconfigure-3.2.5.jar:3.2.5]
I also tried:
package com.example;
import org.springframework.boot.autoconfigure.liquibase.LiquibaseConnectionDetails;
import org.springframework.boot.docker.compose.core.RunningService;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionDetailsFactory;
import org.springframework.boot.docker.compose.service.connection.DockerComposeConnectionSource;
import org.springframework.boot.docker.compose.service.connection.jdbc.JdbcUrlBuilder;
class CustomLiquibaseConnectionDetailsFactory2
extends DockerComposeConnectionDetailsFactory<LiquibaseConnectionDetails> {
CustomLiquibaseConnectionDetailsFactory2() {
super("custom-postgres");
}
@Override
protected LiquibaseConnectionDetails getDockerComposeConnectionDetails(
DockerComposeConnectionSource source) {
return new CustomLiquibaseConnectionDetails(source.getRunningService());
}
private static class CustomLiquibaseConnectionDetails implements LiquibaseConnectionDetails {
private final String jdbcUrl;
CustomLiquibaseConnectionDetails(RunningService service) {
this.jdbcUrl = new JdbcUrlBuilder("postgresql", 5432).build(service, "example");
}
@Override
public String getUsername() {
return "example_ow";
}
@Override
public String getPassword() {
return "example_ow";
}
@Override
public String getJdbcUrl() {
return jdbcUrl;
}
}
}
org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsFactory=\
com.example.CustomDockerComposeConnectionDetailsFactory,\
com.example.CustomLiquibaseConnectionDetailsFactory2
spring:
main:
allow-bean-definition-overriding: true
The application starts.
The result
map contains both custom entries
result = {LinkedHashMap@4873} size = 2
{Class@4893} "interface org.springframework.boot.autoconfigure.liquibase.LiquibaseConnectionDetails" -> {CustomLiquibaseConnectionDetailsFactory2$CustomLiquibaseConnectionDetails@4906}
{Class@4898} "interface org.springframework.boot.autoconfigure.jdbc.JdbcConnectionDetails" -> {CustomDockerComposeConnectionDetailsFactory$CustomDockerComposeConnectionDetails@4907}
Unfortunately, the bean injected into the liquibase
bean is not our custom one but the default one:
connectionDetails = {JdbcAdaptingLiquibaseConnectionDetailsFactory$1@6852}
Implementing Ordered
did not help either.
Should I author a PR adding Ordered
support?
Basically if previous != null
then check for Ordered
and use the one with the lowest priority—if tied throw "duplicate" exception
Thanks for the offer. That's definitely room for improvement here but I'm not yet sure what we should do. Ideally, you wouldn't have to mess around for connection details factories at all to do what you want. We'll discuss it as team and try to figure out what we want to do here.
This won't help with the Liquibase side of things, but for Postgres things can be improved slightly by having the custom JdbcConnectionDetails
implement EnvironmentAware
. It can then retrieve properties from the environment for the username and password:
@Override
public void setEnvironment(Environment environment) {
this.environment = environment;
}
@Override
public String getUsername() {
return this.environment.getProperty("spring.datasource.username");
}
@Override
public String getPassword() {
return this.environment.getProperty("spring.datasource.password");
}
I've experimented a bit in this branch. With no need for a custom connection details factory, it lets you have a compose file like this:
services:
database:
image: postgres:16.3-alpine3.19'
ports:
- '5432'
environment:
- 'POSTGRES_USER=sa'
- 'POSTGRES_PASSWORD=sa'
- 'POSTGRES_DB=mydatabase'
labels:
# $$ is required to disable Docker Compose's own interpolation
# https://docs.docker.com/compose/compose-file/12-interpolation/
- org.springframework.boot.jdbc.username=$${spring.datasource.username}
- org.springframework.boot.jdbc.password=$${spring.datasource.password}
This is very much an experiment and, with this particular approach, I dislike how broad the changes would be to expand support across other service types or even just across the other factories for JdbcConnectionDetails
. Putting that aside for now, I quite like the end result from an external perspective. I haven't yet looked at the Liquibase side of things.
Just one question about this approach:
Where/how do you specify the desired spring profile?
One could have fine-grained profiles containing only the credentials and then run the application with several profiles.
So instead of -Dspring.profiles.active=dev
you might have -Dspring.profiles.active=dev,jpa_dev,liquibase_dev,kafka_dev,something_else_dev
.
You'd specify the profiles as you normally would when starting the app. As usual, the profiles would influence the application properties and YAML files that are loaded into the Spring environment. The placeholders in the label values are then resolved against the environment.
Two alternative ideas:
Label org.springframework.boot.service.use-environment-configuration
or something similarly named.
If true
the Docker Compose support will use the appropriate config from the Spring environment for the annotated service.
It would not be as explicit though.
Label org.springframework.boot.service.configuration-precedence
with values like compose-only
, spring-only
, compose-first
, spring-first
.
Thanks for suggestion.
I'm not sure that Spring Boot's Docker Compose support should be that tightly coupled to the configuration properties that are defined in spring-boot-autoconfigure
. Just for the case of a DataSource's username, it would require the Docker Compose support to know about spring.datasource.username
, spring.datasource.hikari.username
, spring.datasource.dbcp2.username
, etc.
There's also the possibility that the JdbcConnectionDetails
bean has been defined using some other properties. The Docker Compose support would then have no way of knowing what properties to use when overriding it with the JDBC URL from the compose-managed container.
While it may make the compose YAML slightly more verbose, I think it will be better to configure the use of properties explicitly rather than trying to do it automatically. That also allows people who want to hardcode the custom credentials rather than using their application's properties to achieve their goal too.
Spring Boot 3.3.4
I could workaround Liquibase with a custom AutoConfiguration.
By default, by getting the ConnectionDetails
with a custom factory, as seen in this thread, the application used the values from the application property files instead of creating one with the superuser credentials from the Docker container env variables.
But then, LiquibaseAutoConfiguration
would get a JdbcAdaptingLiquibaseConnectionDetailsFactory.LiquibaseConnectionDetails
, that took my custom JdbcConnectionDetails
and just wrapped them to use in Liquibase
as well.
As I wanted a different Liquibase
user, I did this...
src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports
example.CustomLiquibaseAutoConfiguration
src/main/kotlin/example/CustomLiquibaseAutoConfiguration.kt
@AutoConfiguration(before = [LiquibaseAutoConfiguration::class])
@ConditionalOnClass(
LiquibaseAutoConfiguration::class
)
class CustomLiquibaseAutoConfiguration {
@Bean
@Primary
fun customLiquibaseConnectionDetails(): LiquibaseConnectionDetails = CustomLiquibaseConnectionDetails()
private class CustomLiquibaseConnectionDetails :
LiquibaseConnectionDetails, EnvironmentAware {
private lateinit var environment: Environment
override fun setEnvironment(environment: Environment) {
this.environment = environment
}
override fun getUsername() = environment.getProperty("spring.liquibase.user")
override fun getPassword() = environment.getProperty("spring.liquibase.password")
override fun getJdbcUrl() = environment.getProperty("spring.datasource.url")
}
}
Now the Liquibase
auto configuration gets these connection details and I can enjoy different DB users and passwords for the Postgres superuser, my app main datasource and Liquibase.
Spring Boot 3.2.5
The explicitly configured usernames and passwords are not used when using the Docker Compose support:
they should not be overwritten by the one configured in
compose.yaml
:Logs
Setup
application.yaml
compose.yaml
docker/db/init/001-create-users-and-database.sh
docker/db/init/002-create-schema.sh
src/main/resources/db/changelog/db.changelog-master.yaml
I have not verified but I suspect that all
spring.datasource.*.username
andspring.datasource.*.password
properties are affected.