search

timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-62738] Can't connect JNLP agent trhough traefik2 TCP SNI ingressroutetcp #1082

Open timja opened 4 years ago

timja commented 4 years ago

We're trying to connect remote agents to a jenkins master deployed on k8s infra by using JNLP4 protocol. Traefik2 ingressroutetcp is like this:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: jenkins-agent
namespace: default
spec:
entryPoints:

The connection is succesfull if we avoid the host SNI verification but if we want to evaluate the target host the connection never reaches jenkins agent port service. We also tried (-Djsse.enableSNIExtension=true):

java -Djsse.enableSNIExtension=true -jar agent.jar -jnlpUrl http://jenkins.localhost.test:8080/computer/test-agent-tcp/slave-agent.jnlp -secret @secret-file -workDir "/tmp"

but connection is not performed:

Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirJun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /tmp/remoting as a remoting work directoryJun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Both error and output logs will be printed to /tmp/remotingJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: test-agent-tcpJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener INFO: Jenkins agent is running in headless mode.Jun 21, 2020 5:33:10 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 4.3Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /tmp/remoting as a remoting work directoryJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among http://jenkins-test:8080/, http://jenkins.localhost.test:8080/Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability checkJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Agent discovery successful  Agent address: tcp.localhost.test  Agent port:    8081  Identity:      0e:89:a3:be:c8:76:25:b1:3d:36:74:68:9a:a6:63:fbJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: HandshakingJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to tcp.localhost.test:8081Jun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Trying protocol: JNLP4-connectJun 21, 2020 5:33:21 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Protocol JNLP4-connect encountered an unexpected exceptionjava.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:743) at hudson.remoting.Engine.run(Engine.java:518)Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:283) at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164) at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.access$000(AckFilterLayer.java:45) at org.jenkinsci.remoting.protocol.impl.AckFilterLayer$1.run(AckFilterLayer.java:179) at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:964) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) at java.base/java.lang.Thread.run(Thread.java:834)
Jun 21, 2020 5:33:21 PM hudson.remoting.jnlp.Main$CuiListener errorSEVERE: The server rejected the connection: None of the protocols were acceptedjava.lang.Exception: The server rejected the connection: None of the protocols were accepted at hudson.remoting.Engine.onConnectionRejected(Engine.java:828) at hudson.remoting.Engine.innerRun(Engine.java:768) at hudson.remoting.Engine.run(Engine.java:518)

Can you please confirm if agent.jar is already prepared to send SNI information?

Regards

 

Originally reported by danielmurga, imported from: Can't connect JNLP agent trhough traefik2 TCP SNI ingressroutetcp
  • assignee: peppe
  • status: Open
  • priority: Minor
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 4 years ago

danielmurga:

Hi jthompson can you please a look, we've just changed the component to the right one (remoting)

Best regards,

timja commented 4 years ago

jthompson:

[Please note that Jenkins has deprecated the term "slaves" and replaced it with "agents". The replacements are ongoing in various locations. Please use the accepted terms in new usages.]

timja commented 4 years ago

jthompson:

I'm afraid I don't have any information on using SNI with Remoting. I wouldn't have high hopes for it working, though maybe someone has figured it out.

You might try the new WebSockets implementation for agent-initiated connections. As it works over existing HTTPS mechanisms it may provide better results with SNI.

timja commented 4 years ago

alexander_kazakov:

I found the same issue with envoy, dug in and found the actual reason. I think it's applicable to Traefik2 too.

Long story short, Jenkins agent connects to a Jenkins instance over tcp, sends a few bytes with the protocol version it's going to use and upgrades the connection to SSL for JNLP4 protocol after that. Load balancer accepts connection, expects it to be secured (when you enable SNI check), sees that the client sends something unexpected (no ssl handshake at this point, unencrypted data with the protocol version goes first) and rejects the connection. Jenkins agent uses ssl tools provided by the java sdk that fully supports SNI. The problem is that jenkins agent doesn't use SSL in the beginning and if the load balancer doesn't support unsecured tcp connections (SNI check enabled) then the agent cannot connect to the Jenkins instance.

What to do:
1. Allow unsecured connections in load balancer.
2. Use websockets instead of jnlp agents if possible as Jeff suggested.
3. I couldn't use websockets for some reasons so I ended up patching the agent to wrap the connection in a ssl layer as the first step. Not ideal as it encrypts data twice but it works tho. Also jenkins does some weird stuff with certificates (agents get them from the http endpoint and use them to connect to the jnlp port, it doesn't work the jnlp and web port for some reason have different hostnames)