build tasks streamlining and avoidance of file copies

[henrjk]

Currently the build tasks have an output-dir parameter which defaults to docker (relative to working-dir which is another task parameter). This can be used to populate a Docker context.

In many build scripts the build output first populates a build dir and is then copied to the output-dir.

A docker context can also be limited using a .dockerignore. In addition newer (non-daemon) builders will likely not copy unneeded files anyhow (see https://docs.docker.com/build/building/context/).

By avoiding copying build output to a docker context, build tasks do not need parameters anymore to control what is being copied and thus should be simpler.

The proposal is to remove the output-dir parameter.

When build caching is enabled (cache-build parameter), copying is still needed. However for this one would not need a build script parameter anymore. Instead the build task would know where the natural build output is for a specific technology.

This change might also impact the ods-package-image task for example to have docker-dir default to ..

[michaelsauter]

• This will (negatively) affect compatibility with ODS quickstarters, unfortunately
• I am in favour of the change as I agree that output copying is brittle and contributes to confusion

[michaelsauter]

Instead the build task would know where the natural build output is for a specific technology.

I just had a closer look into how that aspects work and worry it isn't that easy. What is the "natural build output"? In theory it can be anywhere so we need a knob to tell the build cache what to retain, no?

[henrjk]

Instead the build task would know where the natural build output is for a specific technology.

I just had a closer look into how that aspects work and worry it isn't that easy. What is the "natural build output"? In theory it can be anywhere so we need a knob to tell the build cache what to retain, no?

It could indeed be that this still needs to be declared for continued support of build caching. I was coming from the npm build where there is already a build-dir.

However a problem is with multi module builds for example with gradle it might produce stuff in multiple build dirs. Perhaps it would make sense to delegate the copying to/from cache to the build script itself. The following sections would have to be replaced with such invocations.

https://github.com/opendevstack/ods-pipeline/blob/ca46bcb944f2a9c108324b15148f156112caf2b0/build/package/scripts/cache-build.sh#L76-L87

and

https://github.com/opendevstack/ods-pipeline/blob/ca46bcb944f2a9c108324b15148f156112caf2b0/build/package/scripts/copy-build-if-cached.sh#L85-L98

The mentioned sections would be replaced by invocation of scripts, perhaps

cp-build-to-cache [-verbose] <cache-dir> cp-cache-to-build [-verbose] <cache-dir>

Which are added to the build tools docker image. For each language these would invoke the build tool with the appropriate target.

[michaelsauter]

Now that you mention the npm task, I think there is some duplication between the build and output dir. Maybe one could live with just one dir to declare, and that would be the dir to cache.

About multiple dirs: maybe it should be Gradle's responsibility to unify them back into a single dir.

[oalyman]

At least for gradle there is no default build output, as this depends on the plugin you use and the multitude of build outputs you then still might have (fat jar, folder structures, zip file). And maybe you want to compose the build output even out of multiple results from multiple plugins etc.