squid won't cache gzipped response

[GoogleCodeExporter]

Using v1.2.0 of ecap adapter with squid 3.1.10 in reverse-proxy mode on linux 
x86_64 (CentOS 5.5) with libecap 0.0.3.  Underlying webserver is setting Vary: 
Accept-Encoding for content meant to be cached and gzipped.

With debug_options set to ALL,9 in squid, we see two problems with caching the 
gzipped response:

1) Squid retains the original content length, even though the ecap adapter 
removes that header.  From squid's cache.log:

 storeEntryValidLength: Checking '1078B4E8EC1D17CFEBCD533EE19F7FD6'
 storeEntryValidLength: object_len = 14843
 storeEntryValidLength: hdr_sz = 360
 storeEntryValidLength: content_length = 20006
 storeEntryValidLength: 5523 bytes too small; '1078B4E8EC1D17CFEBCD533EE19F7FD6'

2) The ecap adapter doesn't respond right on the final call to abContent().  
From squid's cache.log again:

 Adaptation::Ecap::XactionRep will stop, reason: exception

To handle (1), see squid bug 3149 with patch, hopefully to be merged into next 
release - essentially we have to make sure squid resets the content length when 
we add or remove the content-length header from the ecap adapter.  Squid will 
still cache a response with an unspecified content length, which is the case of 
a gzipped response as we can't predict the size of the response when returning 
the headers without finishing the gzipping.  That would require buffering the 
entire response before even returning headers to squid, and that's pretty 
inefficient.  Instead we gzip in pieces just like the adapter already does, but 
make sure squid properly reflects that we don't know the new content length.

The attached patch (for patch -p1) deals with (2) - we have to make sure to 
return properly in the case when sendingAb == opComplete.  The patch also 
corrects a == that should be a = when setting receivingVb, although I don't 
think that affects the cache-ability of the gzipped response.

To test this, set up a similar reverse-proxy situation with the standard v1.2.0 
of the ecap adapter and request an html page with the proper headers for 
caching, and request the exactly same URL twice (expect MISS then HIT, get two 
MISSes).  Note that you could also test with libecap's example modifying 
adapter, doing enough string replacements in there that you end up with a 
shorter content length compared to the original response - that will show (1) 
above, not (2), as the modifying adapter behaves properly for sendingAb == 
opComplete.

Original issue reported on code.google.com by jonathan...@gmail.com on 4 Feb 2011 at 5:57

Attachments:

• gzip-cache.diff

[GoogleCodeExporter]

Original comment by constant...@gmail.com on 12 Feb 2011 at 9:29

• Changed state: Accepted

[GoogleCodeExporter]

Patch has been included in version 1.3.0. Thank you, Jonathan!

Original comment by constant...@gmail.com on 13 Feb 2011 at 2:58

• Changed state: Fixed