Declarative Pipeline: Lock multiple resource types

[rnickle79]

Hi, I would like to lock two resources during the 'Test' stage below. I'm trying to get one resource with the 'SERVER' label, and one resource with the 'CLIENT' label. According to #23 this should possible but I can't nail down the syntax. Any idea how to accomplish this?

 pipeline {
     agent any
     stages {
         stage('Build') {
             steps {
                 echo "Building..."
             }
         }
         stage('Test') {
             options {
                 lock(label: 'SERVER', variable: 'MY_RESOURCE', quantity: 1) 
                 lock(label: 'CLIENT', variable: 'MY_RESOURCE', quantity: 1) 
             }
             steps {
                 echo "Resource Locked: ${env.MY_RESOURCE}"
                 echo "Testing..."
             }
         }
         stage('Deploy') {
             steps {
                 input "Are you ready to deploy?"
                 echo 'Deploying...'
             }
         }
     }
 }

[rnickle79]

Hmmm... it looks like using 'extra' does the trick. However, I noticed my variable is null if the job has to wait at all.

 pipeline {
     agent any
     stages {
         stage('Build') {
             steps {
                 echo "Building..."
             }
         }
         stage('Test') {
             options {
                 lock(extra:[[label: 'SERVER',quantity: 1],[label: 'CLIENT', quantity: 1]], label: 'OTHER_THING', quantity: 1, variable: "MY_RESOURCE")
             }
             steps {
                 echo "Resource Locked: ${env.MY_RESOURCE}"
                 echo "Testing..."
                 sh 'sleep 60'
             }
         }
         stage('Deploy') {
             steps {
                 input "Are you ready to deploy?"
                 echo 'Deploying...'
             }
         }
     }
 }

[bbara]

Hey, not one of the devs or related with the project, but I don't think that the current version will work with declarative pipelines, since they won't let you use duplicated options and the lock syntax does not allow to lock two resources in one call. Therefore, I would suggest you to either use a scripted pipeline or encode the two resources into one "common" (with the disadvantage of losing some flexibility) until someone "allows" to add arrays with multiple resource specifications to the syntax :)

Edit: Oh yeah you are right, just saw the undocumented extra member, so this should already work. Edit2: I tried it with #151 (my current working state) and it works there:

Found 0 available resource(s). Waiting for correct amount: 1.
[{Label: OTHER_THING, Quantity: 1},{Label: SERVER, Quantity: 1},{Label: CLIENT, Quantity: 1},] is locked, waiting...
Lock acquired on [{Label: OTHER_THING, Quantity: 1},{Label: SERVER, Quantity: 1},{Label: CLIENT, Quantity: 1},]
[Pipeline] {
[Pipeline] echo
Resource Locked: Job 2,Job 3,Job 1

[TobiX]

You are probably also hitting https://issues.jenkins-ci.org/browse/JENKINS-43002 ...

[sterys]

I would rather think he hit the https://issues.jenkins-ci.org/browse/JENKINS-50176

[TobiX]

@sterys That too, but the initial problem is a restriction of the declarative syntax, as documented in JENKINS-43002

[famod]

Besides being undocumented, the problem with extra is that it is unclear in which order the locks are being acquired. Furthermore, all locked resource names end up in a single variable (if set).

[TobiX]

@famod It's available in the Snippet Generator (and therefore on https://jenkins.io/doc/pipeline/steps/lockable-resources/#lock-lock-shared-resource), so it's not undocumented.

[famod]

@famod It's available in the Snippet Generator (and therefore on https://jenkins.io/doc/pipeline/steps/lockable-resources/#lock-lock-shared-resource), so it's not undocumented.

Ok, point taken. It is note entirely undocumented. Still, the concrete semantics (order & variable handling) are not described.

[tomasbjerre]

One might also want to know how much time was spent on waiting for different resources. Proposed solution in #185

[baubakg]

Hi it seems to me that the problem with assigning a variable to two different locked resources is still not adressed:

Previous and within a script we can do:

lock(label: 'service1', variable: 'LOCKED_SERVICE1', quantity:1) {
                                lock(label: 'service2', variable: 'LOCKED_SERVICE2', quantity:1) {
               echo "Locked ${LOCKED_SERVICE1}"
                echo "Locked ${LOCKED_SERVICE2}"
                                    }
}

another version that works:

pipeline {
    agent any
    options {
               lock label: 'service1', variable: 'LOCKED_SERVICE1', quantity: 1                    
    }  
    stages {

        stage ("Set Up environment & Check that all is well") {
            options {
                lock label: 'service2', variable: 'LOCKED_SERVICE2', quantity: 1
            }
            steps {
                script {
                        echo "Locked ${LOCKED_SERVICE2}"
                        echo "Locked ${LOCKED_SERVICE1}"
                }   
            }
        }            
    }
}

It seems that in declarative it is not possible to have more than one distinct locked resource per stage, and to be able to access them.

Two locked resources in an option are not allowed, and when defining extra, doesn't allow (to my knowledge) accessing its name

[jimklimov]

176 says the extra keyword is now documented, at least.

209 and #118 may be about similar problems, and #294 addresses a related issue (exposing the several variables separately at least, though their order may still be suffering)

[gaspardpetit]

So, unfortunately #294 will not expose each extra lock to environment variables - the issue is that LockStepResource (representing each extra resource) does not have any support for variable.

I don't mind having a look while I'm working in that area of the code, but to be honest I don't fully grasp the use cases behind the extra var. I usually use declarative pipelines, so if I need multiple locks, I'll just grab them in different blocks (being careful to always lock in the same order everywhere to avoid deadlocks).

Is it possible to describe the typical use case for extra so that I can make sure the fix will work in real life?

[famod]

Is it possible to describe the typical use case for extra so that I can make sure the fix will work in real life?

IIRC, I used extra when locking via the options block.

[gaspardpetit]

thanks - and do you know if there was a reason why extra was made into a parameter instead of having the whole option or lock step take in an list of resource definitions to lock?

[famod]

No, sorry. I wasn't involved when it was added via #87.

[gaspardpetit]

Okay, I stand corrected - I misunderstood the behaviour of extra.

So https://github.com/jenkinsci/lockable-resources-plugin/pull/294 will add a numbered variable for each locked acquired (with extra or normally). But I don't think that helps in your case, you'd want to use a different variable for each extra.

Now the problem I see with that is that you might have the same lock fulfilling multiple requirements. For example:

lock(extra: [[label: 'label1', quantity: 1]], label: 'label2', variable: 'var', quantity: 1) {
  echo "both: ${env.var}" // comma separated list of locks
  echo "first ${env.var0}" // one of the lock
  echo "second: ${env.var1}" // the other lock
}

In the example above, if it turns out that one lock can fulfill both label1 and label2 then you will end up with only one lock reserved. This was counter-intuitive to me, but I imagine it enables some use cases where a single lock can be reused for multiple purposes...

We could go with:

lock(extra: [[label: 'client', quantity: 1, variable:'client'], [label: 'server', variable: 'server', quantity: 1]]) {
  echo "client: ${env.client}"
  echo "server: ${env.server}"
}

And in the case where a single lock can fulfill both labels, we would still assign them to each variable

Let me know if that's an acceptable design and I can work on this