S2S apache httpd proxy not timing out

[mrpatrickwest]

The issue was that httpd processes were not ending. Turning on ExtendedStatus in apache and watching the server-status screen in the browser noted that the hanging processes were waiting to send a response from s2s/proxy.

Determined that ProxyTimeout was NOT set in the apache httpd configuration. So added that to /etc/httpd/conf.d/data.conf. Will continue to monitor.

The fundamental question is why was it hanging trying to send a response. Is it an issue with S2S? Or did someone go to a browser then browse away from it before the request completed so should have timedout.

Just need to keep an eye on that and do better timeouts in S2S and resource cleanup.

[mrpatrickwest]

Setting ProxyTimeout did NOT resolve the issue.

I browse to http://data.deepcarbon.net/browsers/projects.html. Before it has a chance to render all the content I browse away from it (head over to CNN or ESPN or something)

Looking at the httpd processes, a process is now at 100% CPU and just hanging around.

[mrpatrickwest]

Here's the screen shot from the apache server-status display. The offending processes are the ones that say "POST /s2s/proxy HTTP/1.1"

[mrpatrickwest]

Tried to add the following to <Location /s2s> but did NOT work:

SetEnv force-proxy-request-1.0 1 SetEnv proxy-nokeepalive 1

per https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#envsettings

[mrpatrickwest]

Happens with any of the S2S browsers for deepcarbon, not just one of them.

[mrpatrickwest]

problem is the monitoring of the browsers. I've disabled for now and will revisit at a later time. The nagios/icinga checks call out to the pages. The pages return rather quickly, but the javascript continues to run for much longer, running the queries and setting up the widgets. But icinga/nagios doesn't know this. So the request comes back. S2S/javaScript/widgets don't like that and continue to try to send back the response, hanging, and causing the httpd process to spin.

So the problem remains, but I've disabled what's making it happen. I can still make this happen if I browse to one of the s2s browsers and then browse away before the widgets can populate.

[mrpatrickwest]

The problem has slowed significantly, but is still a problem. Just restarted apache httpd again on deepcarbon. And again it's the S2S proxy processes that are the problem.

[zednis]

What are the timeouts on the apache proxy config and the PHP CURL config in utils.php?

[mrpatrickwest]

Haven’t gotten to that yet. Will work on it next as soon as I find an outlet.

On Jul 16, 2014, at 1:00 PM, Stephan Zednik notifications@github.com wrote:

What are the timeouts on the apache proxy config and the PHP CURL config in utils.php?

— Reply to this email directly or view it on GitHub.

[zednis]

Also, when you refer to s2s/proxy are you talking about the S2S proxy servlet or the proxy-pass configuration in apache?

[mrpatrickwest]

proxy pass, not the tomcat servlet. It’s definitely the httpd processes.

With that said, the proxy pass is proxying to the tomcat servlet. So could be that the tomcat servlet is responding, and apache httpd is relaying the response, but doesn’t have anyplace to push it. So I suppose it could be that the s2s tomcat servlet isn’t timing out properly.

On Jul 16, 2014, at 1:01 PM, Stephan Zednik notifications@github.com wrote:

Also, when you refer to s2s/proxy are you talking about the S2S proxy servlet or the proxy-pass configuration in apache?

— Reply to this email directly or view it on GitHub.

[zednis]

@mrpatrickwest is this still an issue after the recent changes to the DCO apache configuration?

[mrpatrickwest]

If we use proxypass in the apache configuration and go to host/s2s it is a problem.

But changing it to go directly to tomcat using host:8080/s2s it's not a problem.

So at this time we are not having any problems. We haven't had to restart httpd on deepcarbon since we made this change.

But we can't always count on the tomcat port being open to the outside and we have to use proxypass in apache to get to the webapp.

[mrpatrickwest]

I wonder if proxy_ajp would work instead of proxypass statements.

[zednis]

I am marking this as 'wont fix' for now as there is not a known issue in the S2S codebase that is causing this issue to occur. If an issue is found in the codebase we can re-open. For the moment I suggest documenting this issue and the workaround on the github repository wiki.