Custom defaults are always ignored

[jomora]

I have set up a repository for custom defaults according to https://github.wdf.sap.corp/ContinuousDelivery/piper-customDefaultConfiguration

It seems that custom defaults are always ignored if they are set:

https://github.com/SAP/jenkins-library/blob/006aaab50d8328936e6a11ccbef822da5c27e0b5/vars/piperExecuteBin.groovy#L125-L128

Build output:

6:30:56  + ./piper whitesourceExecuteScan --defaultConfig .pipeline/custom-defaults.yml --ignoreCustomDefaults
16:30:57  info  whitesourceExecuteScan - Using stageName 'Code Scan' from env variable 'STAGE_NAME'
16:30:57  info  whitesourceExecuteScan - Project config: '.pipeline/config.yaml'
16:30:57  info  whitesourceExecuteScan - Project defaults: '.pipeline/custom-defaults.yml'
16:30:57  info  whitesourceExecuteScan - Ignoring custom defaults from pipeline config
16:30:57  Error: required flag(s) "buildTool" not set
16:30:57  Usage:
16:30:57    piper whitesourceExecuteScan [flags]

Expectation: If I add custom defaults I want them to be taken into account.

[CCFenner]

Ignoring custom defaults from pipeline config

The custom defaults are ignored if set in your config.yaml file. Files provided as described in the repo via step parameter are not affected.

@stippi2 any further comments?

[stippi2]

The log output is just confusing. The problem is that the piper binary, when running in a docker image, may not have access to the custom defaults. However, the Groovy layer will already have downloaded them and persisted them in the .pipeline folder in the workspace. So the Groovy wrapper for each Go step will pass the custom defaults via --defaultConfig ${customDefaults} parameter and then tell the Go layer that it can ignore the custom defaults (because they are passed via other means). So --ignoreCustomDefaults prevents the defaults from being taken into account twice and also prevents errors when the Go step would have no access rights to the resource. We should maybe just remove the log output, but in the command line for the piper binary, it would still appear. Maybe rename the parameter... "customDefaultsAlreadyIncludedInDefaultConfigList"?

[raumB]

@stippi2 With the most current Piper library version my pipelines consuming custom defaults break because apparently custom default parameters are not available at the point in time when they are needed. I noticed this for the post actions when it comes down to send e-mail notifications, for example.

But I have also this issue: my custom defaults provide some configuration for some stage whereas my local pipeline configuration does not need to add or change this central configuration and therefore does not define anything for that same stage. When the pipeline runs this stage is not being executed but ignored. Only when I repeat the central configuration locally the stage is invoked.

This comment is not about to report these issues I encountered but to ask for some guidance about how I could analyze such issues best.

• From your comment I understand, that I need to observe the .pipeline folder in the workspaces of my pipeline. If this folder or the merged configuration file - for whatever reason - does not exist, the custom defaults cannot be consumed.
• What should I monitor during pipeline execution with regard to consumption of custom defaults. Which - maybe not so obvious - portions of the logs would tell me something in this regard?
• Are there any requirements I need to meet to "catch" parameters from custom defaults? Maybe by repeating the stage or step name in the local configuration as an anchor? No, probably this would not be requiered.

[raumB]

Duplicate: https://github.com/SAP/jenkins-library/issues/2272 ?

Maybe the Go step executed in my breaking stage just wants to download the secured custom defaults once more, despite the fact that it is not necessary anymore. My log reads this error message: https://github.com/SAP/jenkins-library/blob/master/cmd/getConfig.go#L46 .

[raumB]

I was now monitoring all the .yml and .yaml files in the .pipeline folder in the workspaces during pipeline execution. Well, all the different configuration sources seem to be there (custom defaults, local configuration, Piper defaults), but is there also one .yml file that would contain the merged configuration? Such a file would be extremely helpful to check if or if not a certain parameter from the different configuration sources was considered or not. But maybe such a file is not being computed nor stashed at all. How could I check during after pipeline execution which overall pipeline configuration is / was being used?

[stippi2]

but is there also one .yml file that would contain the merged configuration?

No, because the "merged" configuration is step-specific and dynamic based on which stage a step runs in. The stage configuration takes precedence over step configuration.

[github-actions[bot]]

Thank you for your contribution! This issue is stale because it has been open 60 days with no activity. In order to keep it open, please remove stale label or add a comment within the next 10 days. If you need a Piper team member to remove the stale label make sure to add @SAP/jenkins-library-team to your comment.