"Script Path" configuration element doesn't handle well undefined variables

[tmeltser]

Describe the bug When running a job for a new branch (in multi branch pipeline) that has a parameter (defined in the declerative pipeline we are running in the job) as the value of the "Script Path" plugin configuration, the build fails since the plugin treats this parameter (name) as a literal, instead of an empty value. Only after we put a constant string "e.g. Jenkinsfile" in the "Script Path" field, and the build complete successfully, from that point on we can replace the constant with the parameter (that is defined in the job's pipeline)

To Reproduce Steps to reproduce the behavior:

1. Define a multibranch job, with a parameter to hold the name of the pipeline to run (e.g. a list with "Jenkinsfile" and "Jenkinsfile2", and assuming you have Jenkinsfile and Jenkinsfile2 in the repo you are working on)
2. Write the the parameter as the value of the "Script Path" plugin configuration (e.g. Script Path -> $TARGET_JENKINSFILE_FILE_NAME)
3. Run the build
4. The build will fail saying it can't find a pipeline named $TARGET_JENKINSFILE_FILE_NAME

Expected behavior As long as the $TARGET_JENKINSFILE_FILE_NAME isn't defined (the first run), treat it as empty and go with the default Jenkinsfile literal Note: Maybe the plugin should inspect this parameter and in case it starts with "$" - try to resolve it (as environment or parameter variable) and if it's undefined, use the "Jenkinsfile" default

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Microsoft Edge
• Version: Version 89.0.774.54 (Official build) (64-bit)

Additional context Jenkins version: 2.263.3 The intended pipeline has the above mentioned parameter defined as:

    parameters {
        choice (
            name: 'TARGET_JENKINSFILE_FILE_NAME',
            choices: infraJenkinsfileFilesNamesListWithNewLineDelimiter,
            description: 'The desired Jenkinsfile to run'
        )

We like the plugin very much and find it very powerful, but this behaviour is very troubling and any assistance would be very much appreciated.

[tmeltser]

Would appreciate any help, as this "glitch" in this well performing and useful plugin, is making our use of it (indeed somewhat sophisticated) very annoying...

[aytuncbeken]

Hi @tmeltser , Thank you for all the details you have provided. Unfortunately, this is possible only if this variable is defined as Global Variable in Jenkins. The reason is, scriptPath is passed to the CpsScmFlowDefinition ( which is not implemented in this plugin, it is part of workflow api) class which handles the clone action, spots the Jenkins file and runs the job.

What I can do is, I can search for the scriptPath ( which is a variable name in this case. Ex: ${JENKINS_FILE} ) in the global variables and replace with the value.

Hope this helps.

Best

[tmeltser]

Hi @aytuncbeken, First - thanks for taking the time to reply on this issue Second - I've already done this and it works too well, the variable indeed gets resolved, but this prevents the variable from getting its value from the parameters section in the pipeline (global variable hides the pipeline parameters). Third - Maybe you can do the following: in case you understand it is a (unresolved) variable (string starts with "$"), just use a default (e.g. "Jenkinsfile"), otherwise continue as usual (I'm assuming that once a build will start the variable will be replaced with a value from the pipeline parameters)? - can you add this functionality?

[aytuncbeken]

Hey, thanks for returning back. I wonder, how did you manage this to work ? Could you please give an example ? Just I wanted to learn.

3. I guess I can do that. Let me understand well, If starts with $ -> Jenkinsfile if not starts with $ -> pass the value

[tmeltser]

Hi, As for the usage - I've leveraged the plugin to allow for a dynamic selection of Jenkinsfile, which is a very powerful ability; if you'll look at my original mail you'll see the parameter I've defined in the pipeline that allows the user (or even more important - an upstream pipeline...) to invoke several pipelines in the same job job (all of which are defined in a remote repo, in our case a shared sub module in Git) - hope that makes it clear (if not let me know and I'll try to add more details) As for the correction - you are correct, and I'll be happy to do a test run for this correction and report back (a small suggestion - when the plugin runs, it would be very useful to have some short texting on what it encountered in the designated field, and what was its manipulation/resolving on it (if at all))

[amiryo]

Hi All, I would be happy to get the solution suggested below in order to be "flexible" when using more than one jenkinsfile in same repo.

Hi @tmeltser , Thank you for all the details you have provided. Unfortunately, this is possible only if this variable is defined as Global Variable in Jenkins. The reason is, scriptPath is passed to the CpsScmFlowDefinition ( which is not implemented in this plugin, it is part of workflow api) class which handles the clone action, spots the Jenkins file and runs the job.

What I can do is, I can search for the scriptPath ( which is a variable name in this case. Ex: ${JENKINS_FILE} ) in the global variables and replace with the value.

Hope this helps.

Best

[nisdonel]

Hello, I have faced with same issue as @tmeltser too, If you can please give priority to this issue I will thank you very much. Thanks a lot

[ghost]

Hi @aytuncbeken, I stumbled upon this issue while looking for a solution to a problem I am facing . I am using multiple jenkinsfiles for different purposes . so far, used interactive user input to provide jenkinsfile name , but now moving into fully automated pipeline with need a solution to run with undefined var as @tmeltser.

looking forward to any progress with this issue . Thank You Leon

[tmeltser]

Hi @aytuncbeken, Any chance I can get the correction we discussed sometime soon? The current behavior is quite annoying (for our use case), and I now see that there are some other parties interested in this adjustment, so it would benefit not only us... Looking forward to getting this correction sometime soon. 10x, Tiran.

[aytuncbeken]

Hi Everyone,

I made some tests and noticed that currently Jenkins file name can be passed to underlying workflow job which is generated by multibranch job. (As @tmeltser mentioned )

Here how I did it.

Defined Remote Jenkins file provider plugin with a variable name.

Defined, parameters in the Jenkins files which I want to use

So, when I run the job it asks for parameters which I set a different Jenkinsfile name

And it works

[aytuncbeken]

My question is, what is the required feature in this issue ? Processing Global variables and replace if it is defined ?

[tmeltser]

Hi @aytuncbeken, Let me check what you wrote, I'm not sure it works with newly created branches (this is what I have seen), but I'll check again and update back.

Tiran.

[tmeltser]

Hi @aytuncbeken, As I said, the problem is with newly created branches, in the example below I've created a new branch "golan" and as you can see, the plugin doesn't recognizes the variable "TARGET_JENKINSFILE_FILE_NAME" (probably since in this new branch the pipeline never executed and the parameter is unknown):

Branch indexing
Checking out git https://at.mavenir.com/bb/scm/mmp/mcu-infra.git into /var/lib/jenkins/workspace/versionamir_golan@script to read $(TARGET_JENKINSFILE_FILE_NAME)
The recommended git tool is: NONE
using credential BB-create
Cloning the remote Git repository
Cloning repository https://at.mavenir.com/bb/scm/mmp/mcu-infra.git
 > git init /var/lib/jenkins/workspace/versionamir_golan@script # timeout=10
Fetching upstream changes from https://at.mavenir.com/bb/scm/mmp/mcu-infra.git
 > git --version # timeout=10
 > git --version # 'git version 2.26.2'
using GIT_ASKPASS to set credentials Git tag
 > git fetch --tags --force --progress -- https://at.mavenir.com/bb/scm/mmp/mcu-infra.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://at.mavenir.com/bb/scm/mmp/mcu-infra.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision bfb303d636d4fd128fe127460313de248c6495eb (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bfb303d636d4fd128fe127460313de248c6495eb # timeout=10
Commit message: "update"
First time build. Skipping changelog.
 > git --version # timeout=10
 > git --version # 'git version 2.26.2'
ERROR: /var/lib/jenkins/workspace/versionamir_golan@script/$(TARGET_JENKINSFILE_FILE_NAME) not found
Finished: FAILURE

As for the requested change - as you wrote above:

I guess I can do that. Let me understand well,
If starts with $ -> Jenkinsfile
if not starts with $ -> pass the value

As for the if starts with $ -> Jenkinsfile part, if you wish you can add another parameter to hold the default pipeline file name to use in such cases. 10x, Tiran.

[aytuncbeken]

I guess, this is also happening because, It needs to run successfully and set Job properties once.

[tmeltser]

I guess, this is also happening because, It needs to run successfully and set Job properties once.

I agree.

So, is it possible to get the adjustment we have just discussed for a test run?

Tiran.

[tmeltser]

Hi @aytuncbeken, Can you assist us with getting this correction for this long waited (manual) workflow relief? - it will allow full utilization of the plugin (in the non-trivial usage we make of it)? Tiran.

[amiryo]

Hi @aytuncbeken , I will appreciate if I can get the new feature soon, in order to skip manual steps in Jenkins. Every time R&D developers create new feature branch, the pipeline failed since, "Script Path" find the variable as literal (as tmeltser explained)

The new feature required is:
Plugin "Script Path" parameter shall recognize undefined variable even when it is not set for the first time.

How: The implementation you suggested seem correct.

When do you think the fix can be tested and ready ?

Thanks , Amir

[aytuncbeken]

Hi @amiryo and @tmeltser ,

Here is the problem, Processing parameters, which are defined in Jenkinsfile, is beyond the reach of this plugin (It is part of Workflow api). This plugin only gets the ScriptPath variable as string and passes to Workflow api with other parameters ( SCM, Branch, etc...). Therefore, I can not provide solution to this problem.

However, I believe there is a possible solution by using JobDSL. I will look into that.

Sorry for the bad news..

[tmeltser]

Hi @aytuncbeken, I think it is much simpler - the plugin makes use of the "Script Path" parameter, all that is needed is to inspect the value it has before proceeding: If the value begins with "$" - don't use it but rather use the "Jenkinsfile" string as the value of the parameter, Otherwise, use the value as is and procced as usual.

[tmeltser]

Hi @aytuncbeken, Can we finalize this long interaction with a suitable alignment and test it?

[aytuncbeken]

Hi @tmeltser , I checked how can I implement this but this creates another bad situation. What the plugin does, It passes the Jenkinsfile parameter to the underlying Workflow Cps. If I replace $JenkinsFileParam with Jenkinsfile then workflow jobs which are created by multibranch job will always use JenkinsFile.

If this case works I can implement.

[tmeltser]

Hi @aytuncbeken, As far as I can tell (talking abut multibranch jobs), after the first successful build, your plugin will not encounter anymore $JenkinsFileParam but rather the resolved value (e.g., Jenkinsfile), as long as the plugin interferes IFF it encounters a value with a leading $ (by replacing it with hard coded Jenkinsfile). If you could supply us a patch - we would be happy to test it and update. B.T.W - if you wish, you can add a parameter to the plugins GUI, to control this behavior (for example, giving in this new parameter, the desired (pipeline file name) default to be used, in case a leading "$" is encountered as the value).

[tmeltser]

@aytuncbeken - can we have the correction?

[aytuncbeken]

Hi @tmeltser I am going to add on option to the GUI for this behavior which will not be set as default ( This change shouldn't affect other users). Hopefully, I am going to implement next weeks. Thanks

[tmeltser]

Hi @tmeltser I am going to add on option to the GUI for this behavior which will not be set as default ( This change shouldn't affect other users). Hopefully, I am going to implement next weeks. Thanks

10x, looking forward to getting this long waited valuable addition.

[nisdonel]

Hi @aytuncbeken, I'm following your correspondence with Tiran regarding a topic. If you can give it priority I would greatly appreciate it Because we have the same problem with us. Thanks a lot

[ghost]

Hi @aytuncbeken, In reply to ...I am going to add on option to the GUI for this behavior which will not be set as default ( This change shouldn't affect other users). Hopefully, I am going to implement next weeks

Please update on this.. Have you made any progress ? . We are looking forward for this blessed improvement ... Many Thanks.

[aytuncbeken]

HI Everyone,

Thanks for being patient. Hopefully, finishing this week.

[efg770]

Hi @aytuncbeken

We are suffering from the same problem in our Jenkins deployment From what I'm reading here, you are trying to fix that Any ETA for that Fix so I can update my plugin as well?

Thanks Efg770

[aytuncbeken]

Hi Everyone,

Thanks for your patience. Finally managed to implement the feature. So how it works ?

A new checkbox is added to plugin configuration, "Lookup in Parameters for Script Path".

For using this feature,

1. Set script path as variable (any variable name is okay). Ex: ${JenkinsFileParameters}
2. Check the "Lookup in Parameters for Script Path" option
3. Re-Scan MultiBranch Job to apply changes.

So, how this feature works ?

1. Before the the pipeline runs, plugin checks for parameters which are defined in pipeline, and get the values of the Parameters whose name is same with the Script Path
2. If there is a parameter, retrieves its value and passes this as Jenkinsfile path to pipeline
3. If there is no parameter, default Jenkins file name "Jenkinsfile" is passed to pipeline as Jenkinsfile path

An example configuration is below.

1. Defined two Jenkinsfiles (above) which have different echo in it
2. Create a MultiBranch Job which points to above repository in Remote Jenkins File definition
3. Set script path as variable (any variable name is okay). Ex: ${JenkinsFileParam}
4. Check the "Lookup in Parameters for Script Path" option
5. Scan MultiBranch Job to apply changes.

Jenkinsfile

pipeline{
    agent any
    parameters {
      string defaultValue: 'Jenkinsfile', description: '', name: 'JenkinsFileParam', trim: false
      string defaultValue: 'Other', description: '', name: 'Other', trim: false
    }
    stages{
        stage('Test'){
            steps{
                sh "echo"
            }
        }
    }
}

Jenkinsfile2

pipeline{
    agent any
    parameters {
      string defaultValue: 'Jenkinsfile', description: '', name: 'JenkinsFileParam', trim: false
      string defaultValue: 'Other', description: '', name: 'Other', trim: false
    }
    stages{
        stage('Test'){
            steps{
                sh "echo 2"
            }
        }
    }
}

When the repository scanned first time, first Jenkinsfile will be run. On the second time, Pipeline will ask for JenkinsFileParam parameter. If you set this parameter as Jenkinsfile2 , then the second pipeline (Jenkinsfile2) will run.

I hope this fully covers your requirements!

[aytuncbeken]

Before releasing this, I will be glad if one of you can test it. hpi file is uploaded to this message ( please unzip it before using) remote-file.hpi.zip

[tmeltser]

10x @aytuncbeken. Will test the new feature and update you promptly with the results (and the experience).

[tmeltser]

Hi @aytuncbeken, We have tested the new version, and it seems to be working fine! Please go ahead and publish it as a formal release. I'm sure this new ability would be a very robust addition to this plugin (but maybe you should add the use case you wrote above to the plugin documentation, to inspire people to make a good use of the plugin, as we do).

10x, Tiran.

[aytuncbeken]

Happy to hear! I will release it this week hopefully. Documentation may take some time :) If anyone wants to prepare the document and publish, you are very welcome.

[aytuncbeken]

Hi Everyone,

This feature is released with 1.20 version. Hopefully, It will be in the plugin update site shortly.

Additionally, soon I will publish an article about this feature. You can follow me from Linked/Twitter: aytuncbeken

I am closing this ticket. Thank you for all the brain storming!