Commands are not always run in virtualenv

[tsvi]

I have 2 parallel jobs running with the same code. One of them creates a virtualenv, the other one doesn't Both of them run on the same node with at the same time. Both of them ran in a directory named TestRunner@X, with X = 2 failing and X =5 passing.

Directories don't have whitespaces so not #26.

I'm at a loss

Pipeline code:

            steps {
                script{
                    withPythonEnv('python3') {
                        sh "pip install --upgrade --index-url=file:///home/jenkins/pip/simple/ vroom"
                        testString = sh(
                            returnStdout: true,
                            script: "vroom --vroom-db ~/vroom.db create run --weighed-random --session-id ${params.sessionId} --jenkins-id ${env.BUILD_ID}").trim()
                    }

Failing log:

[Pipeline] withPythonEnv
[Pipeline] {
[Pipeline] sh
 > git rev-parse --is-inside-work-tree # timeout=10
 > git config remote.origin.url git://XXXXXXXXX/test_manager.git # timeout=10
Fetching upstream changes from git://XXXXXXXXX/test_manager.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress git://XXXXXXXXX/test_manager.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f aba3f86ccd038d107c664b690845065ed7027ab9
 > git rev-list --no-walk aba3f86ccd038d107c664b690845065ed7027ab9 # timeout=10
+ pip install --upgrade --index-url=file:///home/jenkins/pip/simple/ vroom
Looking in indexes: file:///home/jenkins/pip/simple/
Collecting vroom
Collecting tabulate (from vroom)
Collecting fasteners (from vroom)
Collecting sqlalchemy (from vroom)
Requirement already satisfied, skipping upgrade: six in /usr/lib/python3.6/site-packages (from fasteners->vroom) (1.11.0)
Collecting monotonic>=0.1 (from fasteners->vroom)
Installing collected packages: tabulate, monotonic, fasteners, sqlalchemy, vroom
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/lib/python3.6/site-packages/tabulate.py'
Consider using the `--user` option or check the permissions.

Url 'file:///home/jenkins/pip/simple/pip/' is ignored: it is neither a file nor a directory.
[Pipeline] }
[Pipeline] // withPythonEnv
[Pipeline] }
[Pipeline] // script
[Pipeline] }

Passing log:

[Pipeline] script
[Pipeline] {
[Pipeline] withPythonEnv
 > git config remote.origin.url git://XXXXXXXXXXX/test_manager.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url git://XXXXXXXXXXX/test_manager.git # timeout=10
Fetching upstream changes from git://XXXXXXXXXXX/test_manager.git
using GIT_SSH to set credentials 
 > git fetch --tags --progress git://XXXXXXXXXXX/test_manager.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f aba3f86ccd038d107c664b690845065ed7027ab9
 > git rev-list --no-walk aba3f86ccd038d107c664b690845065ed7027ab9 # timeout=10
$ python3 -m virtualenv --python=python3 /local/jenkins_agent/workspace/TestRunner@5/.pyenv-python3
[Pipeline] {
[Pipeline] sh
+ pip install --upgrade --index-url=file:///home/jenkins/pip/simple/ vroom
Looking in indexes: file:///home/jenkins/pip/simple/
Collecting vroom
Collecting sqlalchemy (from vroom)
Collecting fasteners (from vroom)
Collecting tabulate (from vroom)
Collecting six (from fasteners->vroom)
Collecting monotonic>=0.1 (from fasteners->vroom)
Installing collected packages: sqlalchemy, six, monotonic, fasteners, tabulate, vroom
Successfully installed fasteners-0.14.1 monotonic-1.5 six-1.11.0 sqlalchemy-1.2.14 tabulate-0.8.2 vroom-0.9.2
Url 'file:///home/jenkins/pip/simple/pip/' is ignored: it is neither a file nor a directory.
[Pipeline] sh
+ vroom create run --weighed-random --session-id 230 --jenkins-id 262669
[Pipeline] }

[cstarner]

How are you executing the jobs in parallel? I need to try and recreate this to see what exactly is going on.

[tsvi]

@cstarner For some reason diddn't get notified about your comment.

The following is the code how I start parallel runs (I removed some irrelevant code):

pipeline {
    agent none

    stages {
        stage('Parallel tests') {
            steps {
                script {
                    def parallelRuns = [:]
                    def numberOfRuns = 0
                    def maxNumberOfRuns = 100

                    for (int i = 0; i < 10; i++) {
                        parallelRuns[i] = {
                            waitUntil {
                                build job: 'TestRunner', parameters: [
                                string(name: 'sessionId', value: env.BUILD_ID)
                                ], propagate: false

                                numberOfRuns++
                                return numberOfRuns > maxNumberOfRuns
                            }
                        }

                    }
                    parallel parallelRuns
                }
            }
        }
    }
}

The TestRunner job, starts its own pipeline. The first stage is "Select test" which is done using an in-house python tool (vroom), while trying to upgrade to the latest version we fail a lot of the times as virtualenv is not instantiated.

pipeline {
    agent { label "simulation" }

    stages {
        stage('Select test') {
            steps {
                script{
                    withPythonEnv('python3') {
                        sh "pip install --upgrade --index-url=file:///home/jenkins/pip/simple/ vroom"
                    }
                }
            }
        }
    }
}

[tsvi]

Some more info while debugging this issue.

1. This only happens when we run a lot of executors in parallel (10). When I run 2-3 executors in parallel everything is fine.
2. After adding a logger, I see that in the "bad" cases, even when no virtualenv was created, the plugin reports "Created virtualenv". Looking at the log it seems that the result of capturedOutput https://github.com/jenkinsci/pyenv-pipeline-plugin/blob/d99a4e8c65cda98468c7814ea8ad926f4e19a067/src/main/java/com/github/pyenvpipeline/jenkins/VirtualenvManager.java#L96 is an empty string.
3. In the "bad" cases https://github.com/jenkinsci/pyenv-pipeline-plugin/blob/d99a4e8c65cda98468c7814ea8ad926f4e19a067/src/main/java/com/github/pyenvpipeline/jenkins/VirtualenvManager.java#L111 is never executed.

@cstarner If you can point me on how to add more debugging statements and generate my own version of the plugin, I'd be happy to continue debugging this issue.

Thanks,

[cstarner]

To create more logging statements, at least ones that will show in the run logs, you can user the LOGGER objects (or create your own versions with Logger.getLogger()).

To build your own version of the plugin, clone the repository, and run mvn install -e , (-Dmaven.test.skip=true if you want to build without running the tests; which can take a while). This will generate a .hpi file under the target/ folder.

To use this plugin, open Jenkins and go to Manage Jenkins > Plugin Manager > Advanced. There you will see a section to upload a plugin. Upload the generated hpi file, and it will be installed. It will probably require a reboot of Jenkins to take effect.

Let me know if you have any other questions. If you manage to pin it down, and can fix it, I will take pull requests. If not, let me know your findings, and we'll go from there