Content-Type gets corrupted in multi-threaded setup

[GoogleCodeExporter]

As initially reported by Kyle Alan Hale on modwsgi group
(http://groups.google.com/group/modwsgi/msg/54fee67f6a97428c)
and confirmed later by me, mod_wsgi can sometimes send a corrupted Content-Type 
header, at least on
Windows with the default httpd 2.2.21 binary
(e.g. 
http://mirrors.ircam.fr/pub/apache//httpd/binaries/win32/httpd-2.2.21-win32-x86-
openssl-0.9.8r.msi).

What steps will reproduce the problem?
1. install Trac, the above Apache version, and *any mod_wsgi version* since 
mod_wsgi-2.0c3 (!)
2. in IE9 with the developer tools opened (F12), be sure to select option 
"Cache / Always refresh from server" menu option, then go to the Network tab 
and press Start Capturing
3. open any Trac page, look at the Type column in the Network tab

What is the expected output? What do you see instead?

You most certainly will see at least one "text/css" for one of the .css files 
replaced with some random string, most often ascii ("softspace", "n_field", 
"CGI/1.1", "Trac", etc.), but sometimes also binary garbage.

I've analysed the problem and found a possible "fix" (posted on the mailing 
list and soon to be pushed in a clone). The lines removed in the fix were added 
in b8f578d32018 and indeed when building at that revision or at the closest 
release (mod_wsgi-2.0c3), the problem can be reproduced, while the revision 
just before (00f84b2f89a1) doesn't have the problem.

What remains to be verified:
 - which versions of Apache are affected besides 2.2.21?
 - is it related to some specific APR version or build mode?
 - and most importantly: are we really sure it's a Windows only issue?

Also, I think that Trac is good at exposing this issue because it triggers 
several concurrent requests for the .css files, especially if you have a few 
extra ones added by plugins. In my tests, I have 6 of them (trac.css, 
ticket.css and 4 more coming from the WikiExtrasPlugin 
(http://trac-hacks.org/wiki/WikiExtrasPlugin). It's quite possible that the 
issue is not so apparent if you have only 2 .css files. The Content-Type is 
also sometimes messed up for the .js files requested a bit later, but less 
systematically so. And finally I have a fast machine, maybe that's also a 
factor which helps to uncover the problem ;-)

Original issue reported on code.google.com by chb...@gmail.com on 7 Jan 2012 at 4:31

[GoogleCodeExporter]

Pushed as 6951041fdf52 in the chboos-issue255 clone.

Original comment by chb...@gmail.com on 7 Jan 2012 at 11:42

[GoogleCodeExporter]

(above changeset on the 3.x branch btw)

Also, if it helps, for the Apache 2.2.21 we have:
 - libapr-1.dll version 1.4.5
 - libaprutil-1.dll version 1.3.12

Original comment by chb...@gmail.com on 8 Jan 2012 at 12:01

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I now tested on Linux (SLES11sp1) with the same scripts and config (Apache 
2.2.21 MPM-worker, apr 1.4.5, apr-util 1.3.12, Python 2.7.2, Trac trunk, 
mod_wsgi 3.3) - no issues there.

So definitely Windows specific.

Original comment by chb...@gmail.com on 11 Jan 2012 at 11:19

[GoogleCodeExporter]

This bug can be triggered on linux, so it's not windows specific.
apache2-mpm-worker                2.2.20-1ubuntu1.2
libapache2-mod-wsgi               3.3-2ubuntu3
libapr1                           1.4.5-1
libaprutil1                       1.3.12+dfsg-2
libpython2.7                      2.7.2-5ubuntu1

Original comment by surenkar...@gmail.com on 20 Apr 2012 at 12:47

[GoogleCodeExporter]

@surenkar, it would be interesting to know if my patch 
(http://code.google.com/r/chboos-issue255/source/detail?r=6951041fdf526267018277
822537e9dab3615ea7&name=mod_wsgi-3.X) also fixed the issue for you...

Original comment by christia...@free.fr on 23 Apr 2012 at 7:56

[GoogleCodeExporter]

Stupid Google Code site. I didn't even realise you had found a workaround back 
in January. :-(

Original comment by Graham.Dumpleton@gmail.com on 23 Apr 2012 at 8:56

[GoogleCodeExporter]

Well, stupid Google Groups as well then, as I also posted the patch on the 
mailing list ;-) 
(https://groups.google.com/d/msg/modwsgi/VJkb6rBjaqw/qNd6md6xi3wJ)

Original comment by christia...@free.fr on 23 Apr 2012 at 10:23

[GoogleCodeExporter]

Even more stupid me for missing it twice now as I obviously commented on it in 
the email. That wasn't a good time for me though. I was rushing to get a 
release of work software out before I then went on holidays overseas for 3 
+weeks.

Original comment by Graham.Dumpleton@gmail.com on 23 Apr 2012 at 10:35

[GoogleCodeExporter]

Although I was not able today to reproduce the problem when using the 
mod_wsgi-win32-ap22py27-3.3.so downloaded from here (maybe I didn't try long 
enough?), I got independent confirmation that my patch seems to work (see 
http://trac.edgewall.org/ticket/10675).

Original comment by christia...@free.fr on 23 Apr 2012 at 2:02

[GoogleCodeExporter]

Can you find the code:

                if (*self->config->process_group)
                    r->content_type = apr_pstrdup(r->pool, value);
                else
                    ap_set_content_type(r, value);

and change it to:

                if (*self->config->process_group)
                    r->content_type = apr_pstrdup(r->pool, value);
                else
                    ap_set_content_type(r, apr_pstrdup(r->pool, value));

That may well fix the problem.

Remember to put back stuff to allow threads.

Original comment by Graham.Dumpleton@gmail.com on 23 Apr 2012 at 9:52

[GoogleCodeExporter]

It's driving me crazy... same Apache installation, mod_wsgi.c with the same 
debug code as before, and I can't get it to generate a single buggy value. At 
the time I reported the issue, nearly every request had at least 1, 2 or even 
more problematic values for the content-type.

I even switched back to the rev of Trac which I must have used that day 
(r10902), no difference...

But well, the change you proposed is probably the right one anyway! I've sent a 
binary with that fix to the reporter of the Trac ticket #10675 and I'll report 
back.

Original comment by christia...@free.fr on 26 Apr 2012 at 8:11

[GoogleCodeExporter]

Ok, so my reproduction recipe with the "start capture..." described above 
definitely doesn't work anymore (maybe a change to the IE9 developer tools 
which now serializes the queries?).

But if I simply instrument the mod_wsgi.c with the following function:
{{{
int debug_content_type(request_rec *r)
{
    static int found_issue = 0;
    if (r->content_type && !strstr(r->content_type, "/")) 
        found_issue++;
}
}}}
call that throughout the request life-cycle, then attach to httpd.exe and put a 
breakpoint on the `found_issue++` line, I soon hit it, with the expected 
garbage in content_type. Pfew!

In these testing conditions, I tried your fix, and as expected it solves the 
issue :-)

Original comment by christia...@free.fr on 26 Apr 2012 at 9:17

[GoogleCodeExporter]

Fix for this has been pushed to 3.X branch and trunk for 4.0.

Original comment by Graham.Dumpleton@gmail.com on 7 May 2012 at 3:31

• Changed state: Started
• Added labels: Milestone-Release3.4, Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

For information, I've got the issue with linux / Apache 2.2 / mod_wsgi v3.3. 
But the v3.4 fix the problem.

So it was not windows related.

Original comment by erwan.lo...@gmail.com on 5 Oct 2012 at 12:10

[GoogleCodeExporter]

Was released in 3.4.

Original comment by Graham.Dumpleton@gmail.com on 5 Oct 2012 at 12:18

• Changed state: Fixed