Need to support chunked transfer encoding for request content.

[GoogleCodeExporter]

Code currently uses:

  ap_setup_client_block(r, REQUEST_CHUNKED_ERROR)

as read policy for request content. This means that chunked transfer
encoding isn't supported as there will be no overall content length header.

Ultimately need to support this as WSGI specification requires it. Should
be noted though that very few if any WSGI server adapters currently support
chunked transfer encoding on request content. The only one of those looked
at that tries to support it is CherryPy WSGI server, except that it handles
it by reading all the content up front and then creates a content length
for it. This isn't really a practical solution as it triggers 100 continue
responses before an application may be ready to process data, doesn't allow
streaming of reads by an application thereby avoiding buffering and doesn't
allow end to end communication whereby an application responds to client
data with some response and a client in turn sends more data in response to
the data returned by the application.

As far as implementation goes, issues are that when performing call to
Apache ap_get_client_block() function, the read size must be at least
enough to hold any chunk data length information and any trailer
extensions. This means that when WSGI application uses read() with a size
argument, may need to actually ask for more than the requested amount of
data to satisfy the requirement. We cant return more data than requested,
so will need to buffer internally any extra above what is deemed the
minimum size we can acceptably read.

When application calls read() with no argument, have no idea how much more
data there might be, so can only fall back to reading data in some defined
block size. There should be some minimal default but allow it to be set to
be some higher value with a directive. To make implementation easier, may
make more sense to enforce that this blocked reading be used with read()
even when chunked transfer encoding isn't used. This would be in place of
attempting to read all remaining data based on the content length of the
request. This shouldn't be a problem as a WSGI application should keep
calling read() until an empty string is returned and shouldn't be assuming
that it only has to call it once to have all data returned.

Original issue reported on code.google.com by Graham.Dumpleton@gmail.com on 11 Mar 2007 at 7:23

[GoogleCodeExporter]

With the way that WSGI 1.0 specification is written, an application can't 
actually
support chunked transfer encoding on request content as it is forbidden from 
reading
more than CONTENT_LENGTH. With chunked transfer encoding on request content, 
there is
no CONTENT_LENGTH.

This issue will be looked at again when/if WSGI 2.0 specification is released 
and it
supports chunked transfer encoding on request content.

Original comment by Graham.Dumpleton@gmail.com on 1 Apr 2007 at 1:41

• Added labels: Priority-Low
• Removed labels: Milestone-Release1.0, Priority-High

[GoogleCodeExporter]

Does anything, anywhere, ever actually send chunked HTTP requests? I've never 
seen
it, and for normal applications expecting URL-encoded or multipart/form-data 
request
bodies, I can't see any use for it anyhow. Wouldn't worry too much :-)

Original comment by bobi...@gmail.com on 12 Jun 2007 at 7:52

[GoogleCodeExporter]

Original comment by Graham.Dumpleton@gmail.com on 22 Feb 2008 at 3:25

• Removed labels: Usability

[GoogleCodeExporter]

Hi all.  This bit me today.  Has there been any movement on adding support for
chunked requests?  Without support for chunking, I'll most likely fall back to 
using
PHP rather than python for this portion of my project...

Original comment by dree...@gmail.com on 17 Jul 2008 at 9:37

[GoogleCodeExporter]

Chunked requests are still outside of the WSGI specification. All the same, I 
have been looking at it again in last 
few days because of a separate discussion on mod_python list about how to 
modify mod_python to also support 
chunked request content. I am looking at that and trying to work out how I 
could bring chunked request support 
to mod_wsgi, still for core WSGI functionality be compliant with WSGI, but 
allow people to optionally step outside 
of WSGI if they want to to handle such requests.

So, not forgotten.

Original comment by Graham.Dumpleton@gmail.com on 18 Jul 2008 at 1:47

[GoogleCodeExporter]

Sick of waiting for WSGI folks. In revision 1115 have added ability to handled 
chunked request content. To use 
this you will need to side step the WSGI specification and do stuff it says you 
shouldn't. That is, call read() with 
no arguments to get access to all data, or keep calling read() with a block 
size until it returns the empty string 
which indicates no more data available. Reading until no more data is not 
supposed to be done in WSGI because 
specification says that you should never try to read more data than specified 
by CONTENT_LENGTH. For chunked 
request content we have no CONTENT_LENGTH.

Original comment by Graham.Dumpleton@gmail.com on 16 Nov 2008 at 11:03

• Changed state: Started
• Added labels: Milestone-Release3.0

[GoogleCodeExporter]

Attached 1115.diff including patch. Patch applies cleanly to mod_wsgi 2.X 
source code if want to apply it to that.

Note that if using mod_wsgi 2.3 or older, would also need to apply patch for:

  http://code.google.com/p/modwsgi/issues/detail?id=126

Better off perhaps to use mod_wsgi 2.X branch in subversion as includes the 
latter.

Original comment by Graham.Dumpleton@gmail.com on 7 Mar 2009 at 1:05

Attachments:

• 1115.diff

[GoogleCodeExporter]

Chunked POSTs only seem to work in embedded mode for me (testing 3.0 RC1).  Is 
that a
bug or a limitation?

Original comment by palr...@gmail.com on 1 Jun 2009 at 1:53

[GoogleCodeExporter]

@palrich: There are certain things that can't be done in embedded mode, but 
having absolutely no idea what 
your code is trying to do, I cant say whether you are hitting the one 
limitation I am thinking of.

Please post your actual problems along with small code example that 
demonstrates what you are doing to the 
mod_wsgi mailing list as easier to discuss there than on a ticket.

The mod_wsgi mailing list is on Google Groups at:

  http://groups.google.com/group/modwsgi?hl=en

Original comment by Graham.Dumpleton@gmail.com on 3 Jun 2009 at 7:34

[GoogleCodeExporter]

Version 3.0 of mod_wsgi now released.

Original comment by Graham.Dumpleton@gmail.com on 22 Nov 2009 at 2:47

• Changed state: Fixed