Pre-existing venv not applied on node connected via JNLP agent

[eskuat]

I have pre-created python virtualenv directory on my Jenkins system which I'm trying to use with the following script:

pipeline {
    agent {
        label 'master_jws'
    }
    stages {
        stage('Install requirements') {
            steps {
                withPythonEnv('D:\\venv\\pipeline_posdevelop\\') {
                    bat 'python -msite'
                    bat 'pip uninstall -y tornado'
                    bat 'pip install tornado'
                }
            }
        }
    }
}

and it fails with the following log:

`Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on master_jws in D:\master-jws\workspace\POS\POS pipeline [Pipeline] { [Pipeline] stage [Pipeline] { (Install requirements) [Pipeline] withPythonEnv [Pipeline] { [Pipeline] bat

D:\master-jws\workspace\POS\POS pipeline>python -msite sys.path = [ 'D:\master-jws\workspace\POS\POS pipeline', 'C:\Program Files (x86)\Python36-32\python36.zip', 'C:\Program Files (x86)\Python36-32\DLLs', 'C:\Program Files (x86)\Python36-32\lib', 'C:\Program Files (x86)\Python36-32', 'C:\Program Files (x86)\Python36-32\lib\site-packages', ] USER_BASE: 'C:\Users\Pinogy\AppData\Roaming\Python' (doesn't exist) USER_SITE: 'C:\Users\Pinogy\AppData\Roaming\Python\Python36\site-packages' (doesn't exist) ENABLE_USER_SITE: True [Pipeline] bat

D:\master-jws\workspace\POS\POS pipeline>pip uninstall -y tornado Skipping tornado as it is not installed. You are using pip version 10.0.1, however version 19.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. [Pipeline] bat

D:\master-jws\workspace\POS\POS pipeline>pip install tornado Collecting tornado Using cached https://files.pythonhosted.org/packages/f8/e0/765d7efa0254238a4c8e10b0d82367870fba9b162ff41a0b06967517a68b/tornado-6.0.1-cp36-cp36m-win32.whl Installing collected packages: tornado Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\program files (x86)\python36-32\Lib\site-packages\tornado' Consider using the --user option or check the permissions.

You are using pip version 10.0.1, however version 19.0.3 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. [Pipeline] } [Pipeline] // withPythonEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code 1 Finished: FAILURE`

As you see it tries to use system python not virtualenv 'master_jws' is a node connected with JNLP agent, if I change it to: agent any (letting it run on node provided by Jenkins windows service) then it works fine

My setup: Jenkins: 2.150.3 OS: Windows 10 Pyenv Pipeline Plugin 2.1.1-STAGING Python 3.6.6

[cstarner]

Can you show me the system logs related to this run. I would like to see what it tries to match against

[eskuat]

Can you tell which exact log file you want ? There are too many of them. Here is most relevant portion I've found:

Jun 05, 2019 3:18:12 AM com.github.pyenvpipeline.jenkins.ToolVirtualenv$Factory canBeBuilt
INFO: Found Python ToolDescriptor: jenkins.plugins.shiningpanda.tools.PythonInstallation
Jun 05, 2019 3:18:12 AM com.github.pyenvpipeline.jenkins.ToolVirtualenv$Factory canBeBuilt
INFO: Skipping ToolInstallation: Python3.6
Jun 05, 2019 3:18:12 AM com.github.pyenvpipeline.jenkins.ManagedVirtualenv$Factory verifyExistenceOfFiles
INFO: Checking for the existence of: D:\venv\pipeline_posdevelop\Scripts\activate.bat
Jun 05, 2019 3:18:12 AM com.github.pyenvpipeline.jenkins.ManagedVirtualenv$Factory verifyExistenceOfFiles
INFO: Checking for the existence of: D:\venv\pipeline_posdevelop\Scripts\deactivate.bat
Jun 05, 2019 3:18:14 AM org.jenkinsci.remoting.util.AnonymousClassWarnings warn
WARNING: Attempt to (de-)serialize anonymous class org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController$1 in file:/D:/Jenkins/plugins/durable-task/WEB-INF/lib/durable-task.jar; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/
Jun 05, 2019 3:18:29 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finish
INFO: POS/POS pipeline #40 completed: FAILURE

ps: sorry for taking so long to respond.

[cstarner]

That's quite alright.

If you follow that link that appears in the WARNING about trying to de/serialize an anonymous class, it looks like de/serialization of anonymous classes doesn't play well with remoting. It looks like the issue stems from the durable-tasks plugin. What version are you running? That will help me track this down better

[eskuat]

Durable task plugin version 1.29 is installed

[cstarner]

Ok, so doing a little digging, this issue appears to stem from the durable-tasks plugin, specifically this line, where we have the anonymous class that the warning mentions: https://github.com/jenkinsci/durable-task-plugin/blob/cdad57797c1af6f571baf5c2ab3d7621ad1f8dc0/src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java#L299

Because that anonymous class is used (via getOutput) when diffing the environment variables, the changes to variable brought about to the variables is never captured, and never applied within the step context. This is an issue that needs to be raised with them, and it looks like an issue already exists; however, no one has mentioned any specific harm caused by the bug: https://issues.jenkins-ci.org/browse/JENKINS-55145

To be honest, this looks like a pretty easy fix; that class already has tons of private classes, even a few that implement (or extend?) the same interface / abstract class. Separating it out into it's own named class will almost certainly fix this.

I will be closing this, since this is not a bug related to this plugin. In the mean time, I'd look and see what version you can use that lacks this anonymous class