[JENKINS-52166] Jenkins unresponsive when using native https

[timja]

We are trying to run jenkins using https as described here : https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777

 At first it appeared to work properly, but now the Jenkins web server is unresponsive. Https requests timeout, both when accessing the web pages through a browser and using the http API from other tools.

We tried taking out the https related java parameters and it appears to work with http only.

However, when using both http and https on different ports, Jenkins is unresponsive on both ports.

 

Jenkins does not appear to be running out of memory. It is running dozens of threads but they don't seem to be consuming much CPU time.

Looking at the thread dump does not seem to indicate a deadlock, at least some thread are still running.

Furthermore, we run builds every night and they appear to be running properly. However, since we use http requests to fetch the logs, we have little more then the build result to go on.

Outside of Jenkins, our build is not very stable but has comparable stability to what Jenkins seems to be reporting.

 

Update 1:

Jenkins version is 2.121.1

 

Update 2:

Looking at various logs. It appears that:

 - Native Https was working on May 29, which is when we started using https instead of http.

 - Jenkins was Upgraded from version 2.107.3 to version 2.121.1 on June 07.

 - We observed that Jenkins was unresponsive on June 08.

 

Update 3:

After reverting Jenkins to version  2.107.3,

I can confirm that Native Https support was broken in version 2.121.1.

---

Originally reported by tdelisle, imported from: Jenkins unresponsive when using native https

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

oleg_nenashev:

Ugh, I didn't know that we have documentation for it. I've always thought that we support HTTPS only via external services.
tdelisle which Jenkins version do you use?

[timja]

oleg_nenashev:

The wiki page (https://wiki.jenkins.io/pages/viewpage.action?pageId=135468777) has been created in 2017 by jthornsen and then edited by jpi.

olamy danielbeck WDYT? Should actually support it? Or should we add a kind of disclaimer to the Wiki page ("available but not supported" or so).

[timja]

jthornsen:

I wrote those instructions back when I was required to deploy Jenkins over HTTPS on Windows for one of my customers.  This was on a network without internet access, and they forced me to use Windows, so I figured out a way to make the Jenkins embedded webserver listen over HTTPS on port 443 natively and documented it.  Normally I would just deploy Jenkins on Linux and throw haproxy or nginx or apache httpd in front to handle the SSL.

AFAIK that server had no issues with the web service going down.  It is likely running Jenkins LTS 2.60.3 or maybe something slightly earlier than that.

I do notice in the the Jenkins LTS Changelog that Jenkins LTS 2.73 introduced a new version of Winstone and again in 2.121.1 so it's possible the different Winstone/Jetty versions are causing the issue?

[timja]

oleg_nenashev:

Thanks jthornsen!

Yes, it may be related to Jetty upgrades. That's why I CCed olamy. We do not have test automation for such configuration, so it's hard to say when particularly it became unstable (if it is a wide issue unrelated to the reporter's configuration).

[timja]

tdelisle:

I would like to add that using native HTTPS is much more convenient in our case. 

We run Jenkins on a university network, therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections. 

[timja]

tdelisle:

To add to jthornsen's hypothesis, it appears that the problem appear immediately after upgrading to 2.121.1, and was not present before.

We will try to revert to the previous version and see if the problem goes away.

[timja]

danielbeck:

therefore it seems to me that adding a proxy in front of Jenkins won't prevent insecure connections

https://github.com/jenkinsci/winstone/#command-line-options has httpListenAddress, make that 127.0.0.1.