apache / incubator-pagespeed-ngx

Automatic PageSpeed optimization module for Nginx
http://ngxpagespeed.com/
Apache License 2.0
4.36k stars 363 forks source link

Gzip compression does not work when "combine_css" filter is enabled. #1142

Open deweydb opened 8 years ago

deweydb commented 8 years ago

curl -H 'Accept-encoding: gzip' "https://bid13.com/sites/A.all,,_modules,,_admin_menu,,_admin_menu.css,,q2+default,,_files,,_css,,_css_e95c6ea704b78c593760182162122319.css,Mcc.DBCpIWPb-N.css.pagespeed.cf.HegdJuwYto.css" Gives a plaintext response.

As otto points out that this should just work with no gzip settings at all in nginx: http://serverfault.com/questions/543009/how-do-i-enable-gzip-compression-on-nginx-pagespeed-module-resources#answer-659877

The issue seems to be specific to CSS, as images appear to work fine: curl -H 'Accept-encoding: gzip' "https://bid13.com/sites/default/files/imagecache/product_list/290x165x129-1_0.png.pagespeed.ic.YuxZ0FXqYn.webp"

gives a gzipped response.

My guess is this might have something to do with mime.types, or gzip settings in nginx that i am overlooking. Currently my nginx gzip related settings are:

 # Enable Gzip compressed.
  gzip on;

  # Compression level (1-9).
  #5 is a perfect compromise between size and cpu usage, offering about
  #75% reduction for most ascii files (almost identical to level 9).
  gzip_comp_level    5;

  # Don't compress anything that's already small and unlikely to shrink much
  # if at all (the default is 20 bytes, which is bad as that usually leads to
  # larger files after gzipping).
  gzip_min_length    256;

  # Compress data even for clients that are connecting to us via proxies,
  # identified by the "Via" header (required for CloudFront).
  gzip_proxied       any;

  # Tell proxies to cache both the gzipped and regular version of a resource
  # whenever the client's Accept-Encoding capabilities header varies;
  # Avoids the issue where a non-gzip capable client (which is extremely rare
  # today) would display gibberish if their proxy gave them the gzipped version.
  gzip_vary          on;

  # Compress all output labeled with one of the following MIME-types.
  gzip_types
    application/atom+xml
    application/javascript
    application/json
    application/ld+json
    application/manifest+json 
    application/rss+xml
    application/vnd.geo+json
    application/vnd.ms-fontobject
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xhtml+xml
    application/xml
    font/opentype
    image/bmp
    image/svg+xml
    image/x-icon
    text/cache-manifest
    text/css
    text/plain
    text/vcard
    text/vnd.rim.location.xloc
    text/vtt
    text/x-component
    text/x-cross-domain-policy;
  # text/html is always compressed by HttpGzipModule

and my mime types are:

types {

  # Data interchange

    application/atom+xml                  atom;
    application/json                      json map topojson;
    application/ld+json                   jsonld;
    application/rss+xml                   rss;
    application/vnd.geo+json              geojson;
    application/xml                       rdf xml;

  # JavaScript

    # Normalize to standard type.
    # https://tools.ietf.org/html/rfc4329#section-7.2
    application/javascript                js;

  # Manifest files

    application/manifest+json             webmanifest;
    application/x-web-app-manifest+json   webapp;
    text/cache-manifest                   appcache;

  # Media files

    audio/midi                            mid midi kar;
    audio/mp4                             aac f4a f4b m4a;
    audio/mpeg                            mp3;
    audio/ogg                             oga ogg opus;
    audio/x-realaudio                     ra;
    audio/x-wav                           wav;
    image/bmp                             bmp;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    image/png                             png;
    image/svg+xml                         svg svgz;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/webp                            webp;
    image/x-jng                           jng;
    video/3gpp                            3gp 3gpp;
    video/mp4                             f4p f4v m4v mp4;
    video/mpeg                            mpeg mpg;
    video/ogg                             ogv;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-mng                           mng;
    video/x-ms-asf                        asf asx;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;

    # Serving `.ico` image files with a different media type
    # prevents Internet Explorer from displaying then as images:
    # https://github.com/h5bp/html5-boilerplate/commit/37b5fec090d00f38de64b591bcddcb205aadf8ee

    image/x-icon                          cur ico;

  # Microsoft Office

    application/msword                                                         doc;
    application/vnd.ms-excel                                                   xls;
    application/vnd.ms-powerpoint                                              ppt;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

  # Web fonts

    application/font-woff                 woff;
    application/font-woff2                woff2;
    application/vnd.ms-fontobject         eot;

    # Browsers usually ignore the font media types and simply sniff
    # the bytes to figure out the font type.
    # https://mimesniff.spec.whatwg.org/#matching-a-font-type-pattern
    #
    # However, Blink and WebKit based browsers will show a warning
    # in the console if the following font types are served with any
    # other media types.

    application/x-font-ttf                ttc ttf;
    font/opentype                         otf;

  # Other

    application/java-archive              ear jar war;
    application/mac-binhex40              hqx;
    application/octet-stream              bin deb dll dmg exe img iso msi msm msp safariextz;
    application/pdf                       pdf;
    application/postscript                ai eps ps;
    application/rtf                       rtf;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/vnd.wap.wmlc              wmlc;
    application/x-7z-compressed           7z;
    application/x-bb-appworld             bbaw;
    application/x-bittorrent              torrent;
    application/x-chrome-extension        crx;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-opera-extension         oex;
    application/x-perl                    pl pm;
    application/x-pilot                   pdb prc;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            crt der pem;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xslt+xml                  xsl;
    application/zip                       zip;
    text/css                              css;
    text/html                             htm html shtml;
    text/mathml                           mml;
    text/plain                            txt;
    text/vcard                            vcard vcf;
    text/vnd.rim.location.xloc            xloc;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/vtt                              vtt;
    text/x-component                      htc;

}

Interestingly enough the files stored within ngx_pagespeed cache appear to be gzip encoded.

root@localhost:/etc/nginx# locate Fv5Bfx6ySb
/var/ngx_pagespeed_cache_new/rname/ce_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,
/var/ngx_pagespeed_cache_new/rname/ci_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,
/var/ngx_pagespeed_cache_new/v3/bid13.com/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,

root@localhost:/etc/nginx# file "/var/ngx_pagespeed_cache_new/rname/ce_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,"
/var/ngx_pagespeed_cache_new/rname/ce_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,: data
root@localhost:/etc/nginx# file "/var/ngx_pagespeed_cache_new/rname/ci_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,"
/var/ngx_pagespeed_cache_new/rname/ci_ZWQGdRzuWTaNuemJmWle/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,40,40_,: data
root@localhost:/etc/nginx# file "/var/ngx_pagespeed_cache_new/v3/bid13.com/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,"
/var/ngx_pagespeed_cache_new/v3/bid13.com/https,3A/,2Fbid13.com/sites/A.all,2C,2C_modules,2C,2C_admin_menu,2C,2C_admin_menu.css,2C,2Cq2+default,2C,2C_files,2C,2C_css,2C,2C_css_e95c6ea704b78c593760,-/182162122319.css,2CMcc.lk7tFW8fdx.css.pagespeed.cf.Fv5Bfx6ySb.css,: data
deweydb commented 8 years ago

Update, I disabled the filter: "combine_css" and now the gzip is working correctly for CSS. but it would be nice if i could leave that filter on.

eg: curl -H 'Accept-encoding: gzip' "https://bid13.com/sites/all/modules/admin_menu/A.admin_menu.css,q2.pagespeed.cf.SM_cpBMnRg.css"

returns gzipped content.

oschaaf commented 8 years ago

@deweydb When I add a real user-agent to the requests the css responses do get gzip-encoded:

$ curl -I -X GET "https://bid13.com/sites/A.all,,_modules,,_admin_menu,,_admin_menu.css,,q2+default,,_files,,_css,,_css_e95c6ea704b78c593760182162122319.css,Mcc.DBCpIWPb-N.css.pagespeed.cf.HegdJuwYto.css" -H "pragma: no-cache" -H "accept-encoding: gzip, deflate, sdch" -H "accept-language: en-US,en;q=0.8,nl;q=0.6" -H "upgrade-insecure-requests: 1" -H "user-agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36" -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "cache-control: no-cache" -H "authority: bid13.com" --compressed
HTTP/1.1 200 OK
Server: nginx/1.9.12
...
Content-Encoding: gzip

Querying the same file without specifying a real browser user-agent, I can reproduce that there is no gzip compression applied to the response. @jeffkaufman do you know if this is a design choice around PageSpeed's capability for serving compressed .pagespeed. resources?

I cannot reproduce gzip compression for the webp image you posted, but I think that is a good thing. Gzip compression could potentially make the response for these images larger and would not provide any additional benefits.

deweydb commented 8 years ago

Oh! so that makes a bit of sense then. The issue i ran into is that my CDN is proxying those files, and i don't think the CDN uses a real browser user agent to request from my server.

jeffkaufman commented 8 years ago

When I try this manually I'm getting non-gzipped body with "Content-Encoding: gzip" in the headers:

curl -D- -H 'Accept-encoding: gzip' "https://bid13.com/sites/A.all,,_modules,,_admin_menu,,_admin_menu.css,,q2+default,,_files,,_css,,_css_e95c6ea704b78c593760182162122319.css,Mcc.DBCpIWPb-N.css.pagespeed.cf.HegdJuwYto.css"
Server: nginx/1.9.12
Date: Fri, 04 Mar 2016 18:39:41 GMT
Content-Type: text/css
Content-Length: 33248
Connection: keep-alive
Last-Modified: Fri, 04 Mar 2016 18:17:38 GMT
X-Original-Content-Length: 184868
Content-Encoding: gzip
Vary: Accept-Encoding
Expires: Fri, 04 Mar 2016 18:22:38 GMT
X-Page-Speed: 1.10.33.6-0
Cache-Control: max-age=300,private
X-Proxy-Cache: MISS
Strict-Transport-Security: max-age=31536000

/**
 * Administration Menu.
 *
 * Implementation of Sons of Suckerfish Dropdowns.
 * @see www.htmldog.com/articles/suckerfish
 **/
...
deweydb commented 8 years ago

@jeffkaufman ... right. this is exactly my issue. But i don't know why this only happens when i have the 'combine_css' filter enabled?

jeffkaufman commented 8 years ago

@crowell could you look at this?

crowell commented 8 years ago

sure, i'll try to see what's up