[JENKINS-51587] frequently Windows JNLP agents can't connect to Master

[timja]

frequently Windows JNLP agents can't connect to Master. When it happens we see high CPU load. Only restart of master helps. 

I did not found that specific problem between 6 existing issues that looks the same.

In logs of client:

2018 10:36:39 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted

and

java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Server didn't accept the handshake

in logs of master:

May 29, 2018 12:19:06 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
WARNING: NioChannelHub keys=5 gen=21020468: Computer.threadPoolForRemoting [#8] for BuildVSPro2017-EOD terminated
java.io.IOException: Connection aborted: org.jenkinsci.remoting.nio.NioChannelHub$MonoNioTransport@1816c5ac[name=Channel to /10.0.0.164]

Reconnect of manual JNLP does not helps.

Windows service continues running launched state without crash.

I've attached logs of master and slave before restart.

Here the list of plugins:

ws-cleanup: 0.34
rebuild: 1.25
resource-disposer: 0.8
workflow-multibranch: 2.16
git: 3.5.1
file-operations: 1.7
cloudbees-bitbucket-branch-source: 2.2.10
pipeline-stage-step: 2.3
handy-uri-templates-2-api: 2.1.6-1.0
javadoc: 1.4
envinject: 2.1.3
uno-choice: 1.5.2
blueocean-pipeline-scm-api: 1.2.4
handlebars: 1.1.1
config-file-provider: 2.16.4
ssh-slaves: 1.21
conditional-buildstep: 1.3.6
junit: 1.21
workflow-api: 2.20
docker-commons: 1.8
copyartifact: 1.39
ldap: 1.17
blueocean-pipeline-api-impl: 1.2.4
ivy: 1.27.1
aws-java-sdk: 1.11.264
structs: 1.10
external-monitor-job: 1.7
github: 1.28.0
built-on-column: 1.1
workflow-support: 2.14
blueocean-git-pipeline: 1.2.4
mercurial: 2.1
blueocean: 1.2.4
workflow-cps: 2.40
stashNotifier: 1.11.6
sse-gateway: 1.15
pipeline-input-step: 2.8
workflow-job: 2.11.2
ant: 1.7
artifactory: 2.12.2
cmakebuilder: 2.4.5
github-organization-folder: 1.6
pam-auth: 1.3
skip-certificate-check: 1.0
scm-api: 2.2.2
build-name-setter: 1.6.7
maven-plugin: 2.17
pipeline-stage-view: 2.9
role-strategy: 2.6.0
build-timeout: 1.18
workflow-durable-task-step: 2.15
token-macro: 2.3
jenkins-multijob-plugin: 1.28
pipeline-aws: 1.22
blueocean-dashboard: 1.2.4
schedule-build: 0.4.0
bouncycastle-api: 2.16.2
mailer: 1.20
s3: 0.10.12
multiple-scms: 0.6
branch-api: 2.0.11
mapdb-api: 1.0.9.0
blueocean-bitbucket-pipeline: 1.2.4
build-token-root: 1.4
active-directory: 2.6
node-iterator-api: 1.5.0
email-ext: 2.60
scriptler: 2.9
credentials-binding: 1.13
blueocean-rest-impl: 1.2.4
gradle: 1.27.1
blueocean-personalization: 1.2.4
jackson2-api: 2.7.3
matrix-project: 1.11
xunit: 1.102
purge-job-history: 1.1
blueocean-jwt: 1.2.4
msbuild: 1.27
momentjs: 1.1.1
blueocean-config: 1.2.4
scm-sync-configuration: 0.0.10
windows-slaves: 1.3.1
git-server: 1.7
icon-shim: 2.0.3
parameterized-scheduler: 0.5
nodelabelparameter: 1.7.2
apache-httpcomponents-client-4-api: 4.5.3-2.0
blueocean-events: 1.2.4
pipeline-model-extensions: 1.2
pubsub-light: 1.12
jquery-detached: 1.2.1
run-condition: 1.0
display-url-api: 2.0
blueocean-autofavorite: 1.0.0
workflow-step-api: 2.14
workflow-scm-step: 2.5
credentials: 2.1.16
pipeline-model-declarative-agent: 1.1.1
pipeline-build-step: 2.5.1
htmlpublisher: 1.14
ec2-fleet: 1.1.4
jquery: 1.12.4-0
ace-editor: 1.1
script-security: 1.41
workflow-cps-global-lib: 2.9
parameterized-trigger: 2.35.2
durable-task: 1.14
blueocean-i18n: 1.2.4
pipeline-github-lib: 1.0
subversion: 2.9
plain-credentials: 1.4
antisamy-markup-formatter: 1.5
dynamicparameter: 0.2.0
pipeline-utility-steps: 1.4.1
pipeline-model-api: 1.2
favorite: 2.3.0
timestamper: 1.8.8
jira: 2.4.2
ansicolor: 0.5.2
docker-workflow: 1.13
jira-steps: 1.2.3
blueocean-rest: 1.2.4
github-api: 1.89
pipeline-stage-tags-metadata: 1.2
aws-credentials: 1.23
blueocean-commons: 1.2.4
job-restrictions: 0.6
ec2: 1.37
pipeline-model-definition: 1.2
variant: 1.1
build-env-propagator: 1.0
aws-bucket-credentials: 0.2.2
workflow-aggregator: 2.5
blueocean-web: 1.2.4
pipeline-rest-api: 2.9
blueocean-github-pipeline: 1.2.4
pipeline-milestone-step: 1.3.1
workflow-basic-steps: 2.6
ssh-credentials: 1.13
powershell: 1.3
github-branch-source: 2.2.3
cloudbees-folder: 6.1.2
git-client: 2.5.0
envinject-api: 1.2
parameter-separator: 1.0
blueocean-display-url: 2.1.0
pipeline-graph-analysis: 1.5
authentication-tokens: 1.3
blueocean-pipeline-editor: 1.2.4
metrics: 3.1.2.10
matrix-auth: 1.7
blueocean-jira: 1.2.4
Result: [Plugin:ws-cleanup, Plugin:rebuild, Plugin:resource-disposer, Plugin:workflow-multibranch, Plugin:git, Plugin:file-operations, Plugin:cloudbees-bitbucket-branch-source, Plugin:pipeline-stage-step, Plugin:handy-uri-templates-2-api, Plugin:javadoc, Plugin:envinject, Plugin:uno-choice, Plugin:blueocean-pipeline-scm-api, Plugin:handlebars, Plugin:config-file-provider, Plugin:ssh-slaves, Plugin:conditional-buildstep, Plugin:junit, Plugin:workflow-api, Plugin:docker-commons, Plugin:copyartifact, Plugin:ldap, Plugin:blueocean-pipeline-api-impl, Plugin:ivy, Plugin:aws-java-sdk, Plugin:structs, Plugin:external-monitor-job, Plugin:github, Plugin:built-on-column, Plugin:workflow-support, Plugin:blueocean-git-pipeline, Plugin:mercurial, Plugin:blueocean, Plugin:workflow-cps, Plugin:stashNotifier, Plugin:sse-gateway, Plugin:pipeline-input-step, Plugin:workflow-job, Plugin:ant, Plugin:artifactory, Plugin:cmakebuilder, Plugin:github-organization-folder, Plugin:pam-auth, Plugin:skip-certificate-check, Plugin:scm-api, Plugin:build-name-setter, Plugin:maven-plugin, Plugin:pipeline-stage-view, Plugin:role-strategy, Plugin:build-timeout, Plugin:workflow-durable-task-step, Plugin:token-macro, Plugin:jenkins-multijob-plugin, Plugin:pipeline-aws, Plugin:blueocean-dashboard, Plugin:schedule-build, Plugin:bouncycastle-api, Plugin:mailer, Plugin:s3, Plugin:multiple-scms, Plugin:branch-api, Plugin:mapdb-api, Plugin:blueocean-bitbucket-pipeline, Plugin:build-token-root, Plugin:active-directory, Plugin:node-iterator-api, Plugin:email-ext, Plugin:scriptler, Plugin:credentials-binding, Plugin:blueocean-rest-impl, Plugin:gradle, Plugin:blueocean-personalization, Plugin:jackson2-api, Plugin:matrix-project, Plugin:xunit, Plugin:purge-job-history, Plugin:blueocean-jwt, Plugin:msbuild, Plugin:momentjs, Plugin:blueocean-config, Plugin:scm-sync-configuration, Plugin:windows-slaves, Plugin:git-server, Plugin:icon-shim, Plugin:parameterized-scheduler, Plugin:nodelabelparameter, Plugin:apache-httpcomponents-client-4-api, Plugin:blueocean-events, Plugin:pipeline-model-extensions, Plugin:pubsub-light, Plugin:jquery-detached, Plugin:run-condition, Plugin:display-url-api, Plugin:blueocean-autofavorite, Plugin:workflow-step-api, Plugin:workflow-scm-step, Plugin:credentials, Plugin:pipeline-model-declarative-agent, Plugin:pipeline-build-step, Plugin:htmlpublisher, Plugin:ec2-fleet, Plugin:jquery, Plugin:ace-editor, Plugin:script-security, Plugin:workflow-cps-global-lib, Plugin:parameterized-trigger, Plugin:durable-task, Plugin:blueocean-i18n, Plugin:pipeline-github-lib, Plugin:subversion, Plugin:plain-credentials, Plugin:antisamy-markup-formatter, Plugin:dynamicparameter, Plugin:pipeline-utility-steps, Plugin:pipeline-model-api, Plugin:favorite, Plugin:timestamper, Plugin:jira, Plugin:ansicolor, Plugin:docker-workflow, Plugin:jira-steps, Plugin:blueocean-rest, Plugin:github-api, Plugin:pipeline-stage-tags-metadata, Plugin:aws-credentials, Plugin:blueocean-commons, Plugin:job-restrictions, Plugin:ec2, Plugin:pipeline-model-definition, Plugin:variant, Plugin:build-env-propagator, Plugin:aws-bucket-credentials, Plugin:workflow-aggregator, Plugin:blueocean-web, Plugin:pipeline-rest-api, Plugin:blueocean-github-pipeline, Plugin:pipeline-milestone-step, Plugin:workflow-basic-steps, Plugin:ssh-credentials, Plugin:powershell, Plugin:github-branch-source, Plugin:cloudbees-folder, Plugin:git-client, Plugin:envinject-api, Plugin:parameter-separator, Plugin:blueocean-display-url, Plugin:pipeline-graph-analysis, Plugin:authentication-tokens, Plugin:blueocean-pipeline-editor, Plugin:metrics, Plugin:matrix-auth, Plugin:blueocean-jira]

Thank you in advance,

Ilya

---

Originally reported by ilyag, imported from: frequently Windows JNLP agents can't connect to Master

• status: Closed
• priority: Blocker
• resolution: Cannot Reproduce
• resolved: 2018-10-29T21:30:46+00:00
• imported: 2022/01/10

[timja]

rtyler:

Moving this to JENKINS rather than INFRA, the latter of which is only for Jenkins project infrastructure tasks.

I'm not sure if this is a real bug or configuration error however :-/

[timja]

ilyag:

No problem... When I had tried to open the request I wanted to choose "Jenkins", but there was only Jenkins(website) option.

And I'm also sure that the source of the problem is in correct configuration, but anyway I did not found the correct working configuration.

One more thing (not related to JNLP, but adding information on connectivity) - automatically scaled agents (AWS ec2 windows slaves) in such situation continue to work (with other problems not relevant to the context of this issue).

[timja]

marius_balteanu:

Sorry, I pushed the wrong button.

[timja]

jthompson:

ilyag, you say that there was a high CPU load. Was that on the master? An overloaded system can certainly experience a number of different problems, including slow response causing a loss of connection. Have you been able to determine anything about the cause of that high load? Is it the Jenkins master process itself? Or something else on that system? If it is the Jenkins server, then you should see if you can determine what it is doing and what is causing the load. Perhaps specific jobs, maintenance, or plugins.

The first thing to do is to try and isolate something of the cause. Only after that is it possible to try to figure out what needs to be corrected in which component.

[timja]

jthompson:

The defect report does not provide sufficient information for reproduction or action. The creator has not responded to questions for over two months. Please re-open if you can provide sufficient information.