search

lightbend / cloudflow

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
https://cloudflow.io
Apache License 2.0
321 stars 89 forks source link

buildApp sbt task should build artifacts scoped at the project level #814

Open andreaTP opened 4 years ago

andreaTP commented 4 years ago

Is your feature request related to a problem? Please describe. Running buildApp in an sbt multi-project results in building all the Cloudflow applications contained in the entire build. Reproducible copy pasting this directory: https://github.com/lightbend/cloudflow/tree/master/core/sbt-cloudflow/src/sbt-test/sbt-cloudflow/version setting the cloudflow sbt plugin to a fixed version and running:

helloWorld1/buildApp

results in the build of 2 docker images helloWorld1 and helloWorld2

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes? Sbt plugin

Describe the solution you'd like I understand that we want to validate the blueprint for the entire build but the produced artifacts(docker image/ cr file) should be built only for the specific project. Possibly the two steps of building images and producing CR file can be separated in order to avoid the complexity of having to check the dependencies of each project.

Additional context in CI context is often useful to be able to narrow down the build to the minimum required by the single step.

RayRoestenburg commented 4 years ago

Since the blueprint is verified in a build, it is likely possible to keep track of streamlet <-> sub-project, and then only build images for uses streamlets. All sub-projects must be compiled when building an application since it might use streamlets from any of the sub projects.

RayRoestenburg commented 4 years ago

Note that the current behavior is correct, in that it only builds the app.cr for the specific application, when you build from the sub project. This feature request suggest to only build the images that will be used in a specific app.

thomasschoeftner commented 4 years ago

Right, I had to read twice to actually understand the purpose of this. The deployment descriptor json (CR) is already created only for the current sub-project. - That part works for us!

The container images, however, are published for all sub-projects bearing the CloudflowApplicationPlugin. Most CI environments I've seen so far, try to act on a per-artifact level - in this case per sub-project - in order to keep build times shorter, etc. So the CI does not execute a single buildApp command but multiple times - specifically for each sub-project containing a Cloudflow application.

Looking into our container registry, I can see that elaborate builds which involve all / multiple sub-projects publish the container image yielded by a single sub-project multiple times - always moving the same container tag (based on git revision) to the latest image. The other images are obsolete but still consume space (and yield costs).