Ambiguity about applicationSrcDirs and operational issues

Hi,

From its name, the applicationSrcDirs property might suggest that it contains a list of folders to scan to determine what to build.

In fact, after a few implementation tests and analysis of the logs in --verbose mode, we realize that the applicationSrcDirs property must list the root folder of each Git repository(s) present in the workspace.

The applicationSrcDirs property is used to calculate the :giturl: and :githash: information for each of the Git repositories. If you specify folders inside a Git repository, the :giturl: and :githash: informations are calculated and stored for each of the folders, which creates duplicates in the BuildReport.json file, and creates other problems when using this file.

In our operating context, we don't want an entire Git repository to be considered, only certain folders that we want to be able to list and communicate to zAppbuild.

How to make zAppbuild consider only certain folders in a Git repository?

Similarly, we clone secondary Git Repositories (containing COPYBOOKs) into subfolders of a Git Repository folder (containing COBOL programs). If we indicate in applicationSrcDirs the folder containing the subfolders where the secondary repositories are cloned, then the information :giturl: and :githash: are those of the main repository, (those of the folder of main repository): zAppbuild does not detect secondary Git repositories. You must list each of the subfolders corresponding to each of the cloned secondary Git repositories (and there can be many).

How do I make zAppbuild itself discover cloned Git repositories under a particular folder?

Use cases

How to make zAppbuild consider only /src and /pacbase folders in this Git repository?

How to make zAppbuild discover cloned Git repository (in ENVIRONN and GCL subfolders) in the /.imports folder in this Git repository?

Hi @FALLAI-Denis ,

The zAppBuild property applicationSrcDirs is meant to limit the scope to a subset of the directories in your repository.

So, when running it for applicationSrcDirs=${application}/cobol,${application}/copybook

It indeed stores the hash twice, but limits it to these two directories.

*** Obtaining hash for directory /u/dbehm/test-zapp/dbb-zappbuild/samples/MortgageApplication/cobol
** Setting property :githash:MortgageApplication/cobol : 4b4dafa2bb71094cd7cdf2a3a2872e9416f81807
** Setting property :giturl:MortgageApplication/cobol : https://github.com/dennis-behm/dbb-zappbuild.git
*** Obtaining hash for directory /u/dbehm/test-zapp/dbb-zappbuild/samples/MortgageApplication/copybook
** Setting property :githash:MortgageApplication/copybook : 4b4dafa2bb71094cd7cdf2a3a2872e9416f81807
** Setting property :giturl:MortgageApplication/copybook : https://github.com/dennis-behm/dbb-zappbuild.git
** Writing build report data to /u/dbehm/test-zapp-app-out/mortgageout/build.20221111.022013.020/BuildReport.json
** Writing build report to /u/dbehm/test-zapp-app-out/mortgageout/build.20221111.022013.020/BuildReport.html
** Updating build result BuildGroup:MortgageApplication-242_preMerge BuildLabel:build.20221111.022013.020
** Build ended at Fri Nov 11 14:20:34 GMT+01:00 2022

Also impactBuild is working with it. What type of issue do you see there?

When you would like to manage multiple git repositories as the build scope, zAppBuid can manage that. Please have a look to the publication about build scopes at https://www.ibm.com/support/pages/node/6381788

But zAppBuild does not magically understand all git repos under the specified root directory for --workspace. This is configured by the applicationSrcDirs. So, one option might be to parse your .imports configuration - where you have specified the different included repos. Btw - is this .imports your own invention or follows a particular standard?

Dennis

Hi @dennis-behm

What type of issue do you see there?

We create a manifest file (in part) from the BuildReport.json file, which must correspond to the contents of the components package taken from the main git repository (consolidation of impactsBuilds to produce a complete package).. Due to the duplicates on :giturl: and :githash: this caused us several problems:

in the manifest file we want to indicate the reference of the main repository (the one that contains the programs), and we don't need the information on the secondary repositories (the ones that contain the copybooks), but we still need to have a way to distinguish them
if you indicate several folders in the main repository, to filter the content of the main repository, you end up with as many :giturl: and :githash:, all with the same value, and you must only take one of them consideration

applicationSrcDirs=${application}/src,${application}/pacbase,${application}/.imports/ENVIRONN,${application}/.imports/GCL

{                                                                                                                           
    "id": "DBB.BuildResultProperties",                                                                                      
    "type": "PROPERTIES",                                                                                                   
    "properties": {                                                                                                         
        ":githash:ceab3-environn-outinfr-central-test/.imports/GCL": "da96cca4dc8ae0325f4b6f46257d646bb15256ca",            
        "filesProcessed": "7",                                                                                              
        ":giturl:ceab3-environn-outinfr-central-test/pacbase": "https://xxxxxx/scm/CEH49-CENTRAL-ENVIRONN-OUTINFR/ceab3-environn-outinfr-central-test.git",  
        "buildResult": "CLEAN",                                                                                             
        ":githash:ceab3-environn-outinfr-central-test/.imports/ENVIRONN": "c4ef1816f05ddf08eab6ef5a4ec0dd054c6fc758",       
        ":giturl:ceab3-environn-outinfr-central-test/.imports/ENVIRONN": "https://xxxxxx/scm/ceh49-central-environn-outinfr/ceab3-environn-outinfr-central-test-ressources.git", 
        ":githash:ceab3-environn-outinfr-central-test/src": "f2c3f2dcf613709802c2b3a4c932c3b2f681533f",                     
        ":giturl:ceab3-environn-outinfr-central-test/.imports/GCL": "https://xxxxxx/scm/cegcl-central-gcl-outinfr/ceq66-gcl-outinfr-central-ressources.git",
        "fullBuild": "true",                                                                                                
        ":githash:ceab3-environn-outinfr-central-test/pacbase": "f2c3f2dcf613709802c2b3a4c932c3b2f681533f",                 
        ":giturl:ceab3-environn-outinfr-central-test/src": "https://xxxxxx/scm/CEH49-CENTRAL-ENVIRONN-OUTINFR/ceab3-environn-outinfr-central-test.git"   
    }                                                                                                                       
}

We have as many :giturl: and :githash: for the main repository, ceab3-environn-outinfr-central-test.git, as there are folders referenced in applicationSrcDirs, here twice.

Example manifest file to produce (here from a fullBuild): note that the COMMIT information, line type 15, is false because we did not select the correct :githash: (we took that of a secondary repository of COPYBOOKs and not that of the main repository).

11 GIT      PROJET            CEH49-CENTRAL-ENVIRONN-OUTINFR                                                                        
12 GIT      DEPOT             ceab3-environn-outinfr-central-test                                                                   
13 GIT      BRANCHE           master                                                                                                
14 GIT      SERVER            https://xxxxxxxx/scm/CEH49-CENTRAL-ENVIRONN-OUTINFR/ceab3-environn-outinfr-central-test.git    
15 GIT      COMMIT            https://xxxxxxxx/projects/CEH49-CENTRAL-ENVIRONN-OUTINFR/repos/ceab3-environn-outinfr-central-test/commits/da96cca4dc8ae0325f4b6f46257d646bb15256ca
21 BUILD    ID                build.20221114.110821.008                                                                             
22 BUILD    URL               https://yyyyyyyy/dbb/rest/buildResult/12072                                     
23 BUILD    SOURCES           00000007                                                                                              
24 BUILD    TYPE              fullBuild        

89 SYSTEM   TYPE     CODE     PATH                                         HORODATAGEBUILD TYPE_SRC   CODE_SRC CHANGE   RC_COMPIL  HORODATAGE_SRC  PROCESSOR
89 X(8)     X(8)     X(8)     X(44)                                        X(15)           X(10)      X(8)     X(8)     X(10)      X(15)           X(8)     

90 SIRIS    DBRM     S9TST1   SIRIS/DBRM                                   20221114-110821 COBOL      S9TST1   T00001   0004       20221110-144328 COBSTD00 
90 SIRIS    LOADBT   S9TST1   SIRIS/LOADBT                                 20221114-110821 COBOL      S9TST1   T00001   0004       20221110-144328 COBSTD00 
90 SIRIS    DBRM     S9TST2   SIRIS/DBRM                                   20221114-110821 COBOL      S9TST2   T00001   0004       20221110-181829 COBSTD00 
90 SIRIS    LOADBT   S9TST2   SIRIS/LOADBT                                 20221114-110821 COBOL      S9TST2   T00001   0004       20221110-181829 COBSTD00 
90 SIRIS    DBRM     S9TST3   SIRIS/DBRM                                   20221114-110821 COBOL      S9TST3   T00001   0004       20221003-103009 COBSTD00 
90 SIRIS    LOADBT   S9TST3   SIRIS/LOADBT                                 20221114-110821 COBOL      S9TST3   T00001   0004       20221003-103009 COBSTD00 
90 SIRIS    DBRM     S9TST5   SIRIS/DBRM                                   20221114-110821 COBOL      S9TST5   T00001   0004       20220915-170333 COBSTD00 
90 SIRIS    LOADBT   S9TST5   SIRIS/LOADBT                                 20221114-110821 COBOL      S9TST5   T00001   0004       20220915-170333 COBSTD00 
90 HUB      DBRM     S9TSTA   HUB/DBRM                                     20221114-110821 COBOL      S9TSTA   T00001   0004       20220909-170851 COBSTD00 
90 HUB      LOADBT   S9TSTA   HUB/LOADBT                                   20221114-110821 COBOL      S9TSTA   T00001   0004       20220909-170851 COBSTD00 
90 HUB      DBRM     S9TSTB   HUB/DBRM                                     20221114-110821 COBOL      S9TSTB   T00001   0004       20221003-103009 COBSTD00 
90 HUB      LOADBT   S9TSTB   HUB/LOADBT                                   20221114-110821 COBOL      S9TSTB   T00001   0004       20221003-103009 COBSTD00 
90 HUB      DBRM     S9TSTC   HUB/DBRM                                     20221114-110821 COBOL      S9TSTC   T00001   0004       20221003-103009 COBSTD00 
90 HUB      LOADBT   S9TSTC   HUB/LOADBT                                   20221114-110821 COBOL      S9TSTC   T00001   0004       20221003-103009 COBSTD00 

99 SIRIS    DBRM              00000004                                                                                              
99 SIRIS    LOADBT            00000004                                                                                              
99 HUB      DBRM              00000003                                                                                              
99 HUB      LOADBT            00000003

So, one option might be to parse your .imports configuration - where you have specified the different included repos. Btw - is this .imports your own invention or follows a particular standard?

This is a proprietary solution that we have implemented but looks a lot like a solution described in Managing the build scope in IBM DBB builds with IBM zAppBuild. The difference is that we don't want to carry the import information through the Jenkinsfile for the following reasons:

the people of the development project team (of which I am not a member) are responsible for declaring the dependencies between the main repository of the programs and the secondary repositories of the copybooks: they do not have to intervene on the technical configuration of the constructions , especially on the Jenkinsfile.
the people of the tools and procedures administration team (of which I am a member) are responsible for the Jenkins pipeline and the implementation of IBM Dependency Based Build: they do not have to manage the declarations of dependencies between the programs and copybooks

Our Jenkinsfile is reduced to its simplest form (and everything is managed in a Sharedlibrary):

@Library('LibrairieAtelierCentral') _ 
buildCentral(mode:"dbb", typeBuild:'impactBuild');

We can switch from an impactBuild to a fullBuild and vice versa by modifying the typeBuild parameter of the buildCentral function.

Hi,

Could using the --impactScope as described in the Wiki, Use Whitelists in impactBuild scenario, be an alternative to listing the folders to consider rather than the root of the main Git repository?

Is the --impactScope, or something equivalent, usable for a --fullBuild? Even generalized to all build modes?

From which versions of zAppbuild and IBM DBB is this feature available?

Since the organization of our Git repositories is standardized and they all use the same folder organization, can the impactScopeList property be set at the server configuration level?

That said, a quick analysis of the proposed solution shows that the filtering by the scope occurs after the construction of the buildlist... It seems to me preferable that the filtering by the scope intervenes at the time of the construction of the buildlist, by filtering the folders / files to be analyzed rather than by eliminating only a posteriori to remove elements from the buildlist. Perhaps this is a solution that we can learn from to filter upstream rather than downstream of the construction of the buildlist.

Thanks.

Hi,

I return to the problems posed by the declaration of the applicationSrcDirs property.

To date, this property is used by zAppbuild to do two things which in my opinion should be independent:

used to declare the folders (and dependent subfolders) that must be scanned to detect the sources to be built ; whether in fullBuild mode or in impactBuild mode ; e.g. COBOL sources
used to declare folders (and dependent subfolders) that should be scanned for changed items that may trigger the construction of other items, but which themselves did not need to be built ; only in impactBuild mode ; e.g. COPYBOOK sources

Knowing that apparently zAppbuild expects that each folder listed inapplicationSrcDirs is the root of a Git Repository, which does not correspond to our usage since we are listing folders inside the Git Repository (subfolders in the Git Repository), which has the consequence of generating as many :giturl: and :githash: information as folders, even though they are all in the same Git Repository.

Note: if the impacting sources are not identified in the applicationSrcDirs property, then the impacts are not detected (the impactSearch property is not sufficient to declare the impacting sources, when we thought it was its role).

From my point of view, these two needs should be handled by two independent properties:

applicationSrcDirs property: list of folders to analyze for the search for sources to build, e.g. COBOL sources ; to use in fullBuild and impactBuild mode
impactingSrcDirs (or other name) property: list of folders to analyze to find sources that may impact other sources, but which themselves are not to be built, e.g. COPYBOOK sources ; to be used only in impactBuild mode

To see if for each of these folders list it would not also be necessary to manage lists of file extensions to be taken into account (positive list) or to be ignored (negative list).

IBM / dbb-zappbuild

Ambiguity about applicationSrcDirs and operational issues #277