HowTo - Maintain and update information about all items built by multiple iterations

[FALLAI-Denis]

Hi,

In our context, we are doing impactBuilds with zAppbuild, and we want to manage complete packages with all built components. We need to keep context information of each build, (deployment targets for each component because we have muti-targets Applications). The collections and the buildResults stored in the metadata strore of DBB are not sufficient and do not meet our needs (especially since the buildResult does not store the logs, but just references to the log files which remain on the USS disk space ).

We therefore want to manage information outside the DBB metadata store and other than by the logs that we do not want to keep "ad vitam aeternam" on the USS file system.

We want to store information in the package that corresponds to the set of built components, according to the following mechanism:

• before build: recovery of the package from the previous iteration (from Artifactory in our case), preventive deletion of the working PDS, and in the event of impactBuild, recreation of these PDS and loading with the components contained in the package (no PDS creation in case of fullBuild), recovery of information representing construction history
• during the build: in impactBuild mode supply (and possibly creation) of the work PDS created at the stage before build (you can also do a fullBuild on demand), update of the information representing the history of the constructions
• after build: creation of a complete package with all the components built (those of the current iteration but also those of previous iterations) as well as a manifest file describing the contents of the package and created from the history of constructions , depositing the package in the package manager (Artifactory)

We need a hierarchical structure for storing our build history, (source element at first level, and outputs components at lower level, and meta datas at each levels). This hierarchical structure would be parsed from a text file to create a Groovy object at the beginning of the buid, the Groovy object would be updated during the build (addition, modification, deletion of source elements and components), then would be serialized in the same text file at the end of the build.

For history file, we can use a JSON format, but YAML is our preference. After some research, it seems that DBB embeds a snakeYaml library, but Apache suggests using the optional groovy.yaml package and its YamelSlurper class.

Questions :

• do you have any advice or comments to make on the proposed procedure?
• what solution would you recommend for managing YAML files?
• do you have examples of similar processes (impactBuild, fullPackage)?

[FALLAI-Denis]

PS:

There is an example of a full package following impactBuild on the site https://github.com/IBM/dbb: https://github.com/IBM/dbb/tree/main/Pipeline/PackageBuildOutputs

What we want to avoid is step 2 which consists of rereading all the logs of all the buildResults. This does not seem to us to be a reliable solution because it is based on the persistence of log files on the USS file system: there is no guarantee that these files will always be available. Instead we want to maintain a cumulative file that describes all builds applied to a branch in a Git Repository (a collection for zAppbuild and DBB). In case of fullBuild, the cumulative file is reset. In case of impacBuild, the cumulated file is updated (addition, modification, deletion). The cumulative file is included in each full package, which makes it possible to start a new full package from any previous full package.

[dennis-behm]

@FALLAI-Denis these are very interesting use cases. Today, zAppBuild does not maintain a manifest file describing the full application state as a file which can be stored in an artifact repository.

I think we discussed in another thread to keep track of build inputs and outputs as a build map, which is something that goes beyond the BuildReport.json file, which we have today. This principle of build maps could be enhanced to describe the entire application manifest file.

Rather than implementing a custom solution using the scripts, it would be a great enhancement, if DBB would provide an API for this. Would you mind to open an idea on the ibm ideas portal for DBB?

FYI - @drbruce-git @jbyibm

[FALLAI-Denis]

Hi,

Note that in our context, the complete process is divided into three scripts which must share informations:

1. retrieving informations and built output components from previous build iterations from the last package created
2. incremental buid by impactBuild, updating informations (manifest and outputs components built from sources)
3. creation of a new package containing the updated informations and all built output components from all impactBuild from a previous fullBuild (or from an empty set if no first fullBuild)

This passing of information is done using files currently stored in the --outDir. This forces us to modify the normal operation of the build.groovy script to share the same folder for storing information, as is done for the userBuild mode (no addition of a sub-folder to the -- outDir received).

Script at stage 2 is reused for userBuild triggered from IBM Z Open Editor.