Allow generating CR json separately from building/publishing container images

[vkorenev]

Is your feature request related to a problem? Please describe.

There are at least 2 use-cases when I'd like to generate new CR file for existing and published container images:

1. Changing app name in order to be able to deploy it to the same cluster but to a different namespace (we use multiple deployments of the same app for testing purposes).
2. Changing something in a blueprint.conf without modifying any servlet code (I was trying to come up with a way to generate non-conflicting topic names for multiple deployments in the same cluster before I realized that this is also configurable in non-blueprint configs).

Describe the solution you'd like

It would be nice to have some separate sbt task to generate CR file which will not try to build or publish any container images.

Describe alternatives you've considered

Commenting out cloudflowDockerRegistry in build.sbt will suppress image publishing to a remote repository. However, images are still built to the local Docker. But the most inconvenient in this case is that the full image name gets changed in the generated CR file: it is no longer prefixed with the registry name. So I have to edit the CR file to replace all image references, which proved to be error prone (see #975).

[andreaTP]

Thanks for filing this issue! Let me try to go through the different aspects of it.

First of all, we do have a "workaround" to separate image building from publishing (although it's not a first-class citizen):

Commenting out cloudflowDockerRegistry in build.sbt will suppress image publishing to a remote repository. However, images are still built to the local Docker. But the most inconvenient in this case is that the full image name gets changed in the generated CR file: it is no longer prefixed with the registry name.

You can set cloudflowDockerRepository to match the image name that you want to be generated:

https://github.com/lightbend/cloudflow/commit/27cd76bb75075d4762346b3e9dc4ac1492af5684#diff-e4acc41e462eea8646f85f01315bd104cce796f19c0eaa314bea11d3a180fa5bR55-R56

Going to the use cases:

• Deploying to a different namespace is something that we want to support in the long run.
• Changing topics configuration can already be entirely done in the configuration (please correct me if I'm missing any edge case)

And yes:

So I have to edit the CR file to replace all image references, which proved to be error prone

this practice is highly discouraged and we don't guarantee any support for it across Cloudflow versions.

Does this answer already cover your current use-case?

[vkorenev]

@andreaTP thank for your reply!

Our build sets both cloudflowDockerRegistry and cloudflowDockerRepository:

cloudflowDockerRegistry in ThisBuild := Some("docker-ei.jfrog.io")
cloudflowDockerRepository in ThisBuild := Some("mw-iv-streams")

This gives me an image name like this: docker-ei.jfrog.io/mw-iv-streams/mw-iv-streams-streamlets.

I checked that moving the registry part to repository actually produces the same image name while disabling publishing:

cloudflowDockerRegistry in ThisBuild := None
cloudflowDockerRepository in ThisBuild := Some("docker-ei.jfrog.io/mw-iv-streams")

However this does not solve all the problems for me. The build is using dynVer plugin to generate the project version, which is copied to the image tag. Hence after modifying any file I'm getting a new image tag in the CR file and still need to edit it.

So, probably, the problem with reusing the images cannot be solved just by adding a separate generateCR task.

Deploying to a different namespace is something that we want to support in the long run.

I'm really looking forward to it! I've added some hacks and workarounds to achieve that, but it would be great to have it working out of the box!

Changing topics configuration can already be entirely done in the configuration (please correct me if I'm missing any edge case)

Yes, I figured that out eventually.

this practice is highly discouraged and we don't guarantee any support for it across Cloudflow versions

I understand that, so I'm not going to introduce anything like that to CI/CD pipeline. However, for some cases it is better alternative to pushing lots of identical images.

[andreaTP]

The build is using dynVer plugin to generate the project version

This is not mandatory and up to you to enable/disable/workaround if it doesn't play well with your workflow.

In Cloudflow repo examples you can always e.g. export CLOUDFLOW_VERSION=xxx to fix the Cloudflow version: https://github.com/lightbend/cloudflow/blob/master/examples/taxi-ride/project/cloudflow-plugins.sbt#L2

[leozilla]

I would like to follow up on this.

In our project we also want to have the ability to create the "final" docker images without pushing them to a docker registry. This indeed works if u set cloudflowDockerRegistry := None, but we dont want to change our build settings depending on if we want to publish or not. It would be nice to have this options available via a sbt Task or Configuration.

Currently I try to introduce a new sbt Task named buildAppLocal in our project which would do exactly that. Imo this functionality (however it is implemented) should go into the cloudflow sbt plugin sooner or later.

What do u think?

[andreaTP]

@leozilla at the moment you have multiple possible workarounds, e.g. you can define sbt commands or command aliases to package "defining settings + running tasks" (as it's done here https://github.com/lightbend/cloudflow/blob/bafe19d3e3b6f3c35c6b5bc752bce8784f6652a8/tools/build.sbt#L182).

We are currently working on a longer-term plan that is very likely going to make this "feature" a byproduct.