search

leejo / cgi-fast

The new home for CGI::Fast, removing it from the original CGI.pm distribution
4 stars 5 forks source link

CGI::Fast & mod_fcgid: read data timeout #14

Closed clgeen closed 8 years ago

clgeen commented 9 years ago

I'm having what appears to be an intermittent problem with an HP-UX Apache Perl based SSL web application that uses CGI::Fast and the module mod_fcgi.

Somewhere between every 5th - 20th http request made I encounter a "read data timeout" error. An example of the error...

"[Thu Sep 03 14:31:19 2015] [warn] [client 166.61.228.11] mod_fcgid: read data timeout in 180 seconds, referer: https://cisweb.courts.state.va.us/SpriteAct/main/533577/a.html"

This can happen during a simple request that typically takes 1 second and also during longer requests that typically take up to maybe 7-8 seconds. While Apache logs the error as stated above, the browser reports the error as a status 500 internal server error.

System details...

HPUX11i-VSE-OE B.11.31.1103 HP-UX Virtual Server Operating Environment hpuxws22Apache B.2.2.15.13 HP-UX Apache-based Web Server perl D.5.8.8.F Perl Programming Language CGI::Fast 1.07

This does not appear to be something that would be solved by increasing the mod_fcgi timeout value. I've experimented with that with no change in results.

I actually have four Perl applications on this particular web server running under the same conditions and they are all experiencing this problem.

I also have a development web server on which the same four applications are running with no errors at all. The major difference between the two web servers is that the development box is not running SSL.

I've done detailed logging to confirm that the timeout is not occurring during a database query.

Any thoughts as to what may be causing the timeout?

Thanks very much for your effort supporting the CGI::Fast module,

Chris Geen Software Developer Supreme Court of VA

leejo commented 9 years ago

Thanks for raising this as a github issue - is there any way you can provide example code and the apache config that is showing the problem? It's going to be very difficult for me to debug this without being able to closely replicate the environment that is showing the problem.

Also - if you could upgrade to the latest version of the module to see if the problem persists that would also be useful.

clgeen commented 9 years ago

Lee,

Thanks very much for your reply. I'm going to do some more testing on my end here first. If I am unable to rule out CGI::Fast and mod_fcgi as part of the problem, then I'll provide you with an account to our system for you to test the application.

In the meantime, I'm attaching a copy of our Apache config file.

(See attached file: dochtpro.httpd.conf)

Have you ever heard of this type of intermittent timeout problem? Any thoughts on a possible cause?

Thanks again Lee,

Chris Geen Software Developer Supreme Court of VA (804) 786-0117

Lee J ---09/04/2015 05:55:08 PM---Thanks for raising this as a github issue - is there any way you can provide example code and the ap

From: Lee J notifications@github.com To: leejo/cgi-fast cgi-fast@noreply.github.com, Cc: clgeen cgeen@courts.state.va.us Date: 09/04/2015 05:55 PM Subject: Re: [cgi-fast] CGI::Fast & mod_fcgid: read data timeout (#14)

Thanks for raising this as a github issue - is there any way you can provide example code and the apache config that is showing the problem? It's going to be very difficult for me to debug this without being able to closely replicate the environment that is showing the problem. — Reply to this email directly or view it on GitHub.

leejo commented 9 years ago

Chris, i see no attachment. If you replied to the e-mail generated from the github issue then github will have removed the attachment. Having access to your test system might be useful for showing me the issue, but ideally this needs to be reduced down to a minimal test case that demonstrates the problem within CGI::Fast.

Thanks!

clgeen commented 9 years ago

Thanks, Lee. I'm working on reducing this down to a minimal test case. That may take some time. I'm also working with the HP-Support services as well so I'll keep you updated on my progress.

I really appreciate your help,

Chris Geen Software Developer Supreme Court of VA (804) 786-0117

Lee J ---09/09/2015 03:48:29 AM---Chris, i see no attachment. If you replied to the e-mail generated from the github issue then github

From: Lee J notifications@github.com To: leejo/cgi-fast cgi-fast@noreply.github.com, Cc: clgeen cgeen@courts.state.va.us Date: 09/09/2015 03:48 AM Subject: Re: [cgi-fast] CGI::Fast & mod_fcgid: read data timeout (#14)

Chris, i see no attachment. If you replied to the e-mail generated from the github issue then github will have removed the attachment. Having access to your test system might be useful for showing me the issue, but ideally this needs to be reduced down to a minimal test case that demonstrates the problem within CGI::Fast. Thanks! — Reply to this email directly or view it on GitHub.

leejo commented 9 years ago

Chris, no problem and happy to help. BTW - if you are running perl5.8.8 then you are vulnerable to this issue: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1667 which was patched in later perls and then back ported to 5.8.9. I would advise updating your systems to 5.8.9 at the minimum.

More information: http://blog.booking.com/hardening-perls-hash-function.html

leejo commented 8 years ago

@clgeen - any progress on this or can i consider it closed? Thanks!

clgeen commented 8 years ago

Hi Lee. In fact, just yesterday we setup a new web server and have installed the the latest version of mod_fcgi along with the perl modules FCGI and CGI::Fast. I have a total of five web applications that are using CGI::Fast. Two of them are working as expected although three of them are reporting the error: "mod_fcgid: can't apply process slot". The problem here is that when the error is reported, there are zero mod_fcgi processes running. This is after an Apache restart. This is a brand new hp-ux web server with the following... hpuxws22Apache B.2.2.29.01 HP-UX Apache-based Web Server perl E.5.8.8.M Perl Programming Language mod_fcgid-2.3.9.tar.gz CGI::Fast 2.10 FCGI 0.77 I'm going to beginning troubleshooting. In the meantime, any suggestions? Thanks

leejo commented 8 years ago

Two of them are working as expected although three of them are reporting the error

Sounds very odd to me - are the servers created from identical builds? Same versions of everything, etc?

clgeen commented 8 years ago

Yes, very odd. This is a virtual host. The OS, Apache, & perl are all downloaded from HP. The perl modules are from CPAN. I'm using svn to establish the application on the new host. Still looking into it.

clgeen commented 8 years ago

ok. The problem applications did not have the executable bit set. Interesting that it would result in that particular error. I'll keep you posted as to whether or not the original problem persists in the new environment. Thanks, Lee.

clgeen commented 8 years ago

Lee,

I now have all the scripts running successfully. I do have some questions regarding Apache tuning for mod_fcgi. I'm going to read through the docs available at http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html to try and get a better understanding. I may have some questions for you afterwards. Do you have any resources that you'd recommend for configuring mod_fcgi?

Thanks again for you help, Lee.

Chris Geen Software Developer Supreme Court of VA (804) 786-0117

Lee J ---10/16/2015 01:52:13 PM---> Two of them are working as expected although three of them are reporting the error Sounds very odd

From: Lee J notifications@github.com To: leejo/cgi-fast cgi-fast@noreply.github.com, Cc: clgeen cgeen@courts.state.va.us Date: 10/16/2015 01:52 PM Subject: Re: [cgi-fast] CGI::Fast & mod_fcgid: read data timeout (#14)

Two of them are working as expected although three of them are reporting the error

Sounds very odd to me - are the servers created from identical builds? Same versions of everything, etc? — Reply to this email directly or view it on GitHub.

leejo commented 8 years ago

None off the top of my head, i'd do a bit of googling for that and it's going to depend on your specific use case(s). Watch out for any global variables...

clgeen commented 8 years ago

ok, thanks. I'm currently looking at http://www.fastcgi.com/devkit/doc/fastcgi-prog-guide/ch1intro.htm

Thanks,

Chris Geen Software Developer Supreme Court of VA (804) 786-0117

Lee J ---10/20/2015 10:17:47 AM---None off the top of my head, i'd do a bit of googling for that and it's going to depend on your spec

From: Lee J notifications@github.com To: leejo/cgi-fast cgi-fast@noreply.github.com, Cc: clgeen cgeen@courts.state.va.us Date: 10/20/2015 10:17 AM Subject: Re: [cgi-fast] CGI::Fast & mod_fcgid: read data timeout (#14)

None off the top of my head, i'd do a bit of googling for that and it's going to depend on your specific use case(s). Watch out for any global variables... — Reply to this email directly or view it on GitHub.

leejo commented 8 years ago

Bump! Any progress on this?

clgeen commented 8 years ago

Thanks for checking in. We've established a new web server with upgrades across the board and all seems to be operating normally.

Thanks,

Chris Geen Software Developer Supreme Court of VA (804) 786-2178

Lee J ---11/19/2015 04:54:57 AM---Bump! Any progress on this? ---

From: Lee J notifications@github.com To: leejo/cgi-fast cgi-fast@noreply.github.com, Cc: clgeen cgeen@courts.state.va.us Date: 11/19/2015 04:54 AM Subject: Re: [cgi-fast] CGI::Fast & mod_fcgid: read data timeout (#14)

Bump! Any progress on this? — Reply to this email directly or view it on GitHub.

leejo commented 8 years ago

Good news! Sounds like a Heisenbug...

Closing, Thanks!