Application-centric configuration

[aguibert]

Goal

Optimize configuration for the one-app-per-server model and allow all configuration to run a Liberty app to be embedded inside of the application itself, potentially eliminating the need to have a server.xml at all.

Proposal A (properties based)

Allow applications to use microprofile-config.properties to define liberty configuration such as:

liberty.features=jaxrs-2.0,jpa-2.1,mpConfig-1.3
liberty.contextRoot=/myapp
liberty.dataSource.myDS.jndiName=jdbc/myDS
liberty.dataSource.myDS.properties.databaseName=memory:db
liberty.dataSource.myDS.properties.createDatabase=create

To run such an application, we could update the bin/server script to support a new mode:

bin/server runApp /path/to/app.war

When this new entry point is taken, the Liberty kernel would automatically load an extra feature module (e.g. appConfig-1.0) which would contain the necessary logic for looking inside the war file for a microprofile-config.properties and parsing out the relevant Liberty configuration and subsequently bootstrapping the application.

Proposal B (XML based)

Allow applications to embed server.xml configuration inside their applications (e.g. WEB-INF/lib/META-INF/liberty/server.xml), so the application can be more of a portable self-contained unit.

For example, config for a simple REST + JPA application might look like this:

<server>
  <featureManager>
    <feature>jaxrs-2.0</feature>
    <feature>jpa-2.1</feature>
  </featureManager>

  <library id="derbyLib">
    <!-- The app.lib.dir would be a new config variable that would reference the WEB-INF/lib/ dir of the current application -->
    <file name="${app.lib.dir}/derby.jar"/>
  </library>

  <dataSource id="myDS" jndiName="jdbc/myDB">
    <jdbcDriver libraryRef="derbyLib"/>
    <properties.derby.embedded databaseName="memory:db" createDatabase="create"/>
  </dataSource>
</server>

In order to process configuration embedded in an application (including the enablement of additional features), the minimum set of features enabled would be kernel + app manager.

Alasdair brought up that this idea has been discussed before, but was not pursued for two reasons:

1. Starting the application and then processing included configuration would likely lead to restarting the application containing the config, thus resulting in an infinite loop of app restarting
2. It is not valid for all configuration to be embedded, for example, it would not make sense for embedded configuration to define another <application>

I believe both of these issues can be avoided with the following approaches (respectively):

1. Treat embedded configuration as if it was <include>ed in the primary server.xml. Specifically, discover and process embedded configuration before the application is started.
2. Have a new metatype attribute such as ibm:embeddable="true|false" (default=false) that could be defined on metatype <OCD> to indicate whether or not config may be embedded. If non-embeddable config is specified in embedded config, have the config runtime raise an error.

[brenthdaniel]

Some points of clarification and issues raised in today's call:

1) Regarding ordering, there is an assumed ordering of application configurations based on their order in server.xml. Dropins will use the same approach as configDropins (alphabetical.)

2) Lifecycle -- The application lifecycle does not affect the configuration at all. Stopping an application does not remove configuration. Removing an application would result in configuration being removed.

3) Alternative implementation -- By far the easiest way to implement this would be to allow elements to reference application binaries. This gives us the merge behavior control for free and prevents us from having to have knowledge in the config runtime of all different application types (webApplication, osgiApplication, etc, as well as any stack product types that might be defined.)

Remaining issues:

1) Regarding the alternative implementation, it wouldn't allow for using this with dropins (at least not without explicitly referencing a dropins binary in server.xml.)

2) Ordering/resolution: Do application elements act like includes, where an application at the top of xml would have its configuration overridden by config later in the file and an application at the end of the file would take precedence over earlier config? Would dropins come before or after server.xml, or would it be configurable (configDropins has both.)

3) Would a configuration error in the app configuration prevent the application from loading? (assuming no, given the separation of lifecycles, but customers could expect this.)

4) If not using the alternative implementation, we would need a way to prevent app level config from being used. It would have to be another hard coded config option (given that all of this processing has to happen at parsing time.)

5) If not using the alternative implementation we would have to disallow this for application types from stack products. Anyone could extend without our knowledge.

[aguibert]

I think the alternative implementation would defeat the purpose of this proposal, since my intent was to essentially eliminate the need to have a server.xml at ${server.config.dir}/server.xml (and instead just have a boiler plate server.xml that enables a single feature that enables this config/app scanning).

The remaining issues that were not specific to the alternative implementation are:

(2) Ordering/resolution: Do application elements act like includes, where an application at the top of xml would have its configuration overridden by config later in the file and an application at the end of the file would take precedence over earlier config? Would dropins come before or after server.xml, or would it be configurable (configDropins has both.)

I think we should follow the same precedent as configDropins/default/ (i.e. no override). To modify this behavior, we could add a new attribute to the config manager element such as:

<config overrideEmbeddedConfig="true|false"/> <!-- default value of false -->

(3) Would a configuration error in the app configuration prevent the application from loading? (assuming no, given the separation of lifecycles, but customers could expect this.)

I think a config error should not prevent the app from loading. I'd like to advertise this mechanism as an alternate way of doing configDropins, so we could explain this proposal in terms of configOverrides.

(4) If not using the alternative implementation, we would need a way to prevent app level config from being used. It would have to be another hard coded config option (given that all of this processing has to happen at parsing time.)

I'm not entirely sure what you mean by this, but I assume you are suggesting we want to disallow certain config elements such as <application/> from being specified in an embedded server.xml? In that case, we can add a new attribute to metatype indicating whether or not a given OCD/AD is embeddable such as ibm:embeddable="true|false" (default=true).

(5) If not using the alternative implementation we would have to disallow this for application types from stack products. Anyone could extend without our knowledge.

This could be done easily enough if we gated this functionality on a new feature such as embeddedConfig-1.0 or added a new attribute to the config manager such as

<config allowEmbeddableConfig="true|false"/> <!-- default=true -->

[yeekangc]

@hutchig also noted that it will be good to simplify our starter Dockerfile from 3 steps to 2 i.e. without needing to copy a server.xml.

"Could we have an invisible ‘default’ server.xml that has all features when there is none?"

It will be useful if the server.xml can be embedded or we will start with a "default/invisible/boilerplate server.xml" of some sort if one doesn't physically exist -- the application will start fine in any case, the app may have app-specific config from within itself, and config overrides can be applied if necessary (as usual). An "app-centric" approach.

[aguibert]

Alasdair pointed out that that all of our non-kernel Docker images do carry their respective template server.xml as a default server.xml. For example, the webProfile8 image contains a default server.xml that enables webProfile-8.0

[aguibert]

As a potentially more-implementable option, I think we should consider allowing the embedding of a .properties file inside of an application at a well-known location. It might look something like this:

application.properties

liberty.features=servlet-4.0,jpa-2.1,mpConfig-1.3
liberty.dataSource.jdbcUrl=jdbc:db2:etc....

This would effectively eliminate the need for a server.xml or server directory in most configuration scenarios, allowing us to achieve an "application centric" way of configuring and running an application on Liberty.

Instead of launching an app with the usual bin/server run myServer, we could add a new bin/app script that would allow us to launch an application directly, without having a server created. For example:

bin/app run /path/to/myApp.war

It would also be interesting to see if we could create a thin "launcher" jar that would allow us to start an application using java -jar instead of bin/app run

[tjwatson]

I'm not sure why application.properties approach is more implementable. If we have a new launch script app that looks for some liberty configuration files then why not allow the liberty server configuration to be embedded directly in the application?

For example, under /META-INF/liberty/server/ where all the normal liberty server configuration files can live, server.xml, bootstrap.properties etc. Then the app launcer would use the files contained in /META-INF/liberty/server/to create the server configuration at runtime. The only additional configuration the app launcher would need to do is configure the application artifact path as an application before starting the server. I'm sure that can be done with some config dropins or similar.

The application.properties introduces a new concept for configuration of Liberty that has to be understood by users, documented and finally mapped into Liberty configuration at runtime similar to how server.xml is. Unless that provides a benefit above and beyond the current server.xml for the user then I would not be for that approach. On the other hand, if application.properties is deemed much better than dealing with XML and the current ways of configuring Liberty then I would argue we should support something like a server.properties for the server configuration as an alternative to server.xml. I would want the app-centric configuration to look as similar as possible to how an server-centric configuration looks.

[aguibert]

I'm not sure why application.properties approach is more implementable. If we have a new launch script app that looks for some liberty configuration files then why not allow the liberty server configuration to be embedded directly in the application?

I don't think the application.properties approach is necessarily more implementable, than embedding an XML config file. Rather, since it may not make sense to allow all server.xml config to be embedded in an app (e.g. a <webApplication> element), it may be more clear from a user to introduce a subset of configuration as key=value properties that is conducive to embedding inside of an app.

The application.properties introduces a new concept for configuration of Liberty that has to be understood by users, documented and finally mapped into Liberty configuration at runtime similar to how server.xml is. Unless that provides a benefit above and beyond the current server.xml for the user then I would not be for that approach.

It is a new concept for liberty users yes, but I think it's a familiar concept for all other enterprise java developers. Aside from this, an app.properties config would provide benefit because it's more concise than the XML equivalent. For example, consider the two equivalent configurations:

server.xml

<server>
  <featureManager>
    <feature>jaxrs-2.1</feature>
  </featureManager>

  <httpEndpoint id="defaultHttpEndpoint" httpPort="9080"/>
</server>

application.properties

liberty.features=jaxrs-2.1
liberty.http.port=9080

[aguibert]

Another benefit of properties-based configuration over XML-based config is that it becomes easier to override values at runtime. For example:

bin/app run /path/to/myapp.war -Dliberty.http.port=9088

To do the equivalent with XML-based configuration, we'd need to define the variable in our configuration like this:

<server>
  <featureManager>
    <feature>jaxrs-2.1</feature>
  </featureManager>

  <variable name="httpPort" defaultValue="9080"/>
  <httpEndpoint id="defaultHttpEndpoint" httpPort="${httpPort}"/>
</server>

[Emily-Jiang]

Can this info be made to microprofile-config.properties? See this twitter feedback: https://twitter.com/konrbk/status/1200449328847962113

[aguibert]

Sure, we could certainly use the standard microprofile-config.properties file (or just MP Config in general) to load this configuration

[sdaschner]

Hi folks, some thoughts (also see my Gist here):

+1 for supporting a properties-based approach. This would greatly lower the threshold especially for newcomers.

Also big +1 for naming the well-known default location application.properties (which makes more sense naming-wise and avoids coupling to a specific tech). As a side note, we might want to think about whether Liberty supports this properties file name as an additional MP ConfigSource in general, like Quarkus does.

Also correct, properties make it more convenient and obvious how to override values at application start.

Another additional point:

Can we make it possible to alias feature names in the properties file without the version that would default to a later feature version in case the current runtime supports multiple ones? E.g. specifying liberty.features=jaxrs,jsonp,cdi instead of jaxrs-2.1,jsonp-1.1,cdi-2.0. Even I who does this all the time in the <features> constantly have to lookup which version is the latest in the spec. Most developers who configure that don't care or don't know and if they do care we still can make it possible by allowing them to be explicit.

[aguibert]

Specifying versionless features seems a bit dangerous, because in order to do something like liberty.features=jaxrs,jsonp,cdi we'd either have to: A. pin the versionless features to a specific version (e.g. cdi==cdi-2.0), which would become out of date as new versions of those features are released B. break zero-migration guarantee if non-versioned features are provided (e.g. today cdi==cdi-2.0 but in a year it points to cdi-3.0)

Specifying versionless features would be convenient, but I don't think the benefit would outweigh the downside of either of the above 2 options.

I do agree with the pain point about not knowing what features you can put though. Right now users can run wlp/bin/productInfo featureInfo to get a print out of all features that are available in their install. However, many users probably do not know this and it's a bit tedious to do since most users don't interact with Liberty installs directly and instead interact with it through our maven/gradle plugins. Perhaps we could add a mvn liberty:listFeatures goal to the maven plugin and similar for gradle?

[sdaschner]

Good point, however, due to the backwards-compatible nature of EE specs I don't see this as this big of an issue, unless an older version is used instead of a newer one.

The use case I'm thinking of is: the user would like to quickly create an app, can pin down a version of Liberty, and would like to have "the latest versions" of EE specs that version supports. If the Liberty version is fixed, the risk of breaks is pretty much minimized.

I see this more as the BOM approach that a few technologies are taken: Pin down an "umbrella" version, specify the bits you want to have ("latest" if multiple are supported), and be happy with that list (if the umbrella version doesn't change).

Yes, I've used featureInfo a few times, but even that is a bit cumbersome to search if you mostly remember "JAX-RS+CDI+JSON-P". IMO same is true of the Maven plugin.

Thoughts?

[aguibert]

due to the backwards-compatible nature of EE specs I don't see this as this big of an issue, unless an older version is used instead of a newer one.

Not all features in Liberty are Java EE specs though, for example MP features do make breaking changes. Also, regardless of whether a feature is backwards-compatible or not, Liberty's zero migration policy only applies to features being held constant and only the Liberty version updating. For example, we made a subtle breaking change between jdbc-4.1 --> jdbc-4.2 to fix up a design point we had changed our mind on.

I have one idea for how we could work around this by making a liberty.featureSet=<liberty-release> property, which could essentially act as the BOM for the liberty.features property.

However, I think that adding a mvn liberty:listFeatures goal with nice and simple output will be much easier for us to implement and also just as user-friendly. I would liken this to the Quarkus mvn quarkus:listExtensions goal which does essentially the same thing and is the way that Quarkus users discover the available "features" without needing to consult external documentation.

[sdaschner]

I like the idea on fixing the features with the featureSet property. However, how would that work differently than just reusing the Liberty version (if defined)?

+1 on the liberty:listFeatures. However, I see that as a slightly different feature (helpful still).

[aguibert]

However, how would that work differently than just reusing the Liberty version (if defined)?

The benefit of having a featureSet property rather than re-using the current liberty version is that it locks the set of available features based on the featureSet version which solves both problems:

• Users do not have to specify the feature version, they just automatically get the latest (at the time of specifying the featureSet property)
• Zero-migration is still gauraunteed

To give a concrete example, the servlet-4.0 feature was introduced in 18.0.0.2, prior to which the newest servlet feature was servlet-3.1. So if a user had:

liberty.featureSet=18.0.0.1
liberty.features=servlet

They would get servlet-3.1, even if this app was migrated to a newer version of Liberty that did have servlet-4.0. This would satisfy Liberty's zero-migration policy.

[sdaschner]

Ok, got it. If the featureSet property gets defaulted to the Liberty version in case it's not set, then I fully +1 that approach :)

[hlhoots]

@NottyCode Can this be closed out? Kernel team discussed, and thought this is no longer needed / desired. Tx.