Automatically add configuration bundles in deployment script

[sallareznov]

Hi,

As far as I know, when deploying a Lagom application for example with ConductR, the only way to attach configuration bundles to application bundles is to manually edit the deployment script generated by the generateInstallationScript task. In a continuous delivery process, in which everything is automated, this is cumbersome.

My question is: do you plan to automate this task in the near future?

I was thinking about a simple way to do so and I came up with this. Currently, in the generated script, there are three commands for every service in the application. For exemple, for a service named my-service, these three following commands will be executed:

echo "Deploying my-service-impl..."
MY_SERVICE_IMPL_BUNDLE_ID=$(conduct load ../my-service-impl/target/bundle/my-service-impl-v1-9396485e14fdaaf04b2942e5dc9569e994fdcf196fd7c97838199077a5d4eee5.zip  --long-ids -q)
conduct run ${MY_SERVICE_IMPL_BUNDLE_ID} --no-wait -q

We could modify the generateInstallationScript task to add a placeholder where the configuration bundle will be provided. Given a service named my_service, the configuration bundle placeholder can be named __MY_SERVICE_IMPL_CONF_BUNDLE__. Therefore, the commands would look like this:

echo "Deploying my-service-impl..."
MY_SERVICE_IMPL_BUNDLE_ID=$(conduct load ../my-service-impl/target/bundle/my-service-impl-v1-9396485e14fdaaf04b2942e5dc9569e994fdcf196fd7c97838199077a5d4eee5.zip __MY_SERVICE_IMPL_CONF_BUNDLE__ --long-ids -q)
conduct run ${MY_SERVICE_IMPL_BUNDLE_ID} --no-wait -q

Thus, with the sed command, the end-user can replace the placeholder and inject the path to the bundle configuration.

Any thoughts about this? Thanks.

[markusjura]

Generally, this is not a bad idea. Unfortunately, it is not so straightforward. If an sbt project contains a bundle-configuration directory then we create a bundle configuration and append the configuration file to the installation script.

Example https://github.com/typesafehub/sbt-conductr/tree/master/sbt-conductr-tester/lagom-scala-bundle/play-service contains a bundle configuration direcory: bundle-configuration/default.

As a result, the generateInstallationScript creates the following lines for the play-service:

echo "Deploying play-service..."
PLAY_SERVICE_BUNDLE_ID=$(conduct load ../play-service/target/bundle/play-service-0.1.0-SNAPSHOT-57c66f4d82e16af60d16f27198191e6f100626a26efe1cb0981dde75e697077e.zip ../play-service/target/bundle-configuration/default-227747f57d343822f0d7fc000f366477ecc70b2a7de23092c6e3eb548d74ef03.zip --long-ids -q)
conduct run ${PLAY_SERVICE_BUNDLE_ID} --no-wait -q

In this case, adding the additional environment variable $PLAY_SERVICE_BUNDLE_CONFIGURATION to the conduct load line will result in a loading error because two bundle configuration files have been specified.

During script creation, we also cannot check if the environment variable has been set and based on that either add the environment variable or the bundle configuration path from the sbt project to it. The reason is that this environment variable might not be set in the environment at which the script is created but on another environment, e.g. CI server

In general, we have not put much love into the generateInstallationScript so far. I am happy to do quick fixes but for bigger changes, we should talk about the general direction. Currently, we are working on a bndl in the conductr-cli that makes it possible to change the configuration on demand based on certain arguments. The benefit of the bndl command is that it is not bound to sbt and it supports different inputs, e.g. bundle file, bundle configuration file, bundle.conf, oci image, docker image.

The bndl command does not create an installation script based on a given project context, though. This is also not in scope for the command because it is part of the conductr-cli which is installed as a python application and therefore is not aware of a project context.

@huntc Thoughts?

[huntc]

I think that when generating the script we should assign the bundle and any configuration to variables. This way, a user may then use sed and other tools more readily to perform any requisite transformations or even leverage existing env vars for the values. Here's an example:

echo "Deploying play-service..."
PLAY_SERVICE_BUNDLE=${PLAY_SERVICE_BUNDLE:-../play-service/target/bundle/play-service-0.1.0-SNAPSHOT-57c66f4d82e16af60d16f27198191e6f100626a26efe1cb0981dde75e697077e.zip}
PLAY_SERVICE_CONFIG=${PLAY_SERVICE_CONFIG:-../play-service/target/bundle-configuration/default-227747f57d343822f0d7fc000f366477ecc70b2a7de23092c6e3eb548d74ef03.zip}
PLAY_SERVICE_BUNDLE_ID=$(conduct load $PLAY_SERVICE_BUNDLE $PLAY_SERVICE_CONFIG --long-ids -q)
conduct run ${PLAY_SERVICE_BUNDLE_ID} --no-wait -q

@sallareznov How does that sound?

[markusjura]

@huntc I like the idea. Happy to implement that once @sallareznov like it as well.

[sallareznov]

Hi, it looks good to me! :) Thank you.

[huntc]

I'm going to close this as I feel that our Continuous Delivery solution deals with it, along with new backup/restore functionality. There will always be a case of having to initially load a bundle along with its prod configuration, but once up, CD will replace updates to the bundle with any existing config in use. Backup/restore allows you to move bundles around across clusters.