Concurrency/load issues

[csdougliss]

Hi,

I have a store that is running amazingly fast (thanks to turpentine etc) for single users. But when I hit with say Gatling and 200 users concurrently it grinds to a halt.

The setup is Varnish Cache 4.0.1 with Nginx + HHVM + Redis Cache + Redis Session. Server setup is 32 cores, 120gb of RAM, 40 GB SSD.

Gatling

va.util.concurrent.TimeoutException: Request timed out to xxx of 60000 ms
16:48:07.807 [WARN ] i.g.h.a.AsyncHandlerActor - Request 'request_1' failed: java.util.concurrent.TimeoutException: Request timed out to xxx of 60000 ms

Varnish

/usr/sbin/varnishd -P /var/run/varnishd.pid -a 0.0.0.0:80 -T 0.0.0.0:6082 -t 120 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,2G -p feature=+esi_ignore_other_elements -p vcc_allow_inline_c=on -p thread_pools=2 -p syslog_cli_traffic=off -p cli_buffer=24000

/etc/varnish/default.vcl

vcl 4.0;
# Nexcess.net Turpentine Extension for Magento
# Copyright (C) 2012  Nexcess.net L.L.C.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
#51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

## Nexcessnet_Turpentine Varnish v4 VCL Template

## Custom C Code

C{
#include <stdlib.h>
#include <stdio.h>
#include <time.h>
#include <pthread.h>

static pthread_mutex_t lrand_mutex = PTHREAD_MUTEX_INITIALIZER;

void generate_uuid(char* buf) {
    pthread_mutex_lock(&lrand_mutex);
    long a = lrand48();
    long b = lrand48();
    long c = lrand48();
    long d = lrand48();
    pthread_mutex_unlock(&lrand_mutex);
    // SID must match this regex for Kount compat /^\w{1,32}$/
    sprintf(buf, "frontend=%08lx%04lx%04lx%04lx%04lx%08lx",
        a,
        b & 0xffff,
        (b & ((long)0x0fff0000) >> 16) | 0x4000,
        (c & 0x0fff) | 0x8000,
        (c & (long)0xffff0000) >> 16,
        d
    );
    return;
}

}C

## Imports

import std;

## Custom VCL Logic

# Vax Custom Include - VCL V4

# Additional includes for logging
C{
#include <stdio.h>
#include <stdlib.h>
#include <syslog.h>
#include <stddef.h>
#include <sys/time.h>
#include <time.h>
}C

sub vcl_recv {
    #   Add X-Request-Start header so we can track queue times in New Relic RPM beginning at Varnish.
    #if (req.restarts == 0) {
        C{
             /*struct timeval detail_time;
            gettimeofday(&detail_time,NULL);
            char start[20];
            sprintf(start, "t=%lu%06lu", detail_time.tv_sec, detail_time.tv_usec);
            static const struct gethdr_s VGC_HDR_REQ_VARNISH_FAKED_SESSION = { HDR_REQ, "\030X-Varnish-Faked-Session:"};
            VRT_SetHdr(ctx, &VGC_HDR_REQ_VARNISH_FAKED_SESSION, "\020X-Request-Start:", start, vrt_magic_string_end);*/
        }C
    #}

    # Bypass registration form
    if (req.url ~ "^/registration/form") {
        return (pass);
    }
}

sub vcl_backend_error {
    if (beresp.status >= 500) {
        C{
            /*FILE *fp;
            char ft[256];
            struct tm *tmp;
            time_t curtime;

            fp = fopen("/var/log/varnish/error_log", "a");
            time(&curtime);
            tmp = localtime(&curtime);
            strftime(ft, 256, "%D - %T", tmp);

            if(fp != NULL) {
                fprintf(fp, "%s: Error (%s) (%s) (%s)\n",
                ft, VRT_r_req_url(sp), VRT_r_obj_response(sp), VRT_r_req_xid(sp));

                fclose(fp);
            } else {
                syslog(LOG_INFO, "Error (%s) (%s) (%s)",
                VRT_r_req_url(sp), VRT_r_obj_response(sp), VRT_r_req_xid(sp));
            }*/
        }C
    }

    synthetic({"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
   <head>
     <title>"} + beresp.status + " " + beresp.reason + {"</title>
     <style type="text/css">
        * {
            margin: 0;
            padding: 0;
        }
        .error-layout { width: 100%; max-width: 1600px; height: 100%; margin: 0 auto; padding: 0; }
        .error-layout .main-container { height: 100%; }
        .error-layout .main { background: url("/errors/default/images/error_background.jpg") no-repeat #fff; min-height: 590px; color: #1D2B33; float: left; }
        .error-layout .col-main { width: 100%; }
        .error-layout .std { margin: 100px 0 0 400px; padding-left: 270px; background: url("/errors/default/images/warning_icon.png") no-repeat 10px 0; height: 200px; }
        .error-layout h3 { font-size: 400%; margin-bottom: 10px; }
        .error-layout p { font-size: 180%; }
        .error-layout .back { float: left; background: url("/skin/frontend/vax/uk/images/icons/left_arrow.png") no-repeat 8px center #1D2B33; height: 18px; padding: 5px 8px 5px 33px; color: #FFF; font-size: 13px; line-height: 18px; text-decoration: none; margin-top: 10px; }
    /* ======================================================================================= */
    </style>
   </head>
   <body>
     <div class="page error-layout">
        <div class="main-container">
            <div class="main">
                <div class="col-main">
                    <div class="std"><h3>Oops, something went wrong</h3>
                        <!--<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
                        <p>"} + beresp.reason + {"</p>
                        <h3>Guru Meditation:</h3>
                        <p>XID: "} + bereq.xid + {"</p>
                        <hr>-->
                        <p><a class="back" title="Go Back" href="javascript: history.go(-1);">Go Back</a></p>
                    </div>
                 </div>
            </div>
      </div>
    </body>
 </html>
 "});
     return (deliver);
}

# customized vcl_deliver to allow use of cross sub-domain cookies
sub vcl_deliver {
    if (req.http.X-Varnish-Faked-Session) {
        # need to set the set-cookie header since we just made it out of thin air
        call generate_session_expires;
        set resp.http.Set-Cookie = req.http.X-Varnish-Faked-Session +
            "; expires=" + resp.http.X-Varnish-Cookie-Expires + "; path=/";
        if (req.http.Host) {
            if (req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
                    # it's a crawler, no need to share cookies
                    set resp.http.Set-Cookie = resp.http.Set-Cookie +
                    "; domain=" + regsub(req.http.Host, ":\d+$", "");
            } else {
                # it's a real user, allow sharing of cookies between support etc
                if(req.http.Host ~ "^(www\.|spares\.|support\.)?vax\.co\.uk$") {
                    set resp.http.Set-Cookie = resp.http.Set-Cookie +
                        "; domain=.vax.co.uk";
                } else {
                    set resp.http.Set-Cookie = resp.http.Set-Cookie +
                        "; domain=" + regsub(req.http.Host, ":\d+$", "");
                }
            }
        }
        set resp.http.Set-Cookie = resp.http.Set-Cookie + "; httponly";
        unset resp.http.X-Varnish-Cookie-Expires;
    }
    if (req.http.X-Varnish-Esi-Method == "ajax" && req.http.X-Varnish-Esi-Access == "private") {
        set resp.http.Cache-Control = "no-cache";
    }
    if (false || client.ip ~ debug_acl) {
        # debugging is on, give some extra info
        set resp.http.X-Varnish-Hits = obj.hits;
        set resp.http.X-Varnish-Esi-Method = req.http.X-Varnish-Esi-Method;
        set resp.http.X-Varnish-Esi-Access = req.http.X-Varnish-Esi-Access;
        set resp.http.X-Varnish-Currency = req.http.X-Varnish-Currency;
        set resp.http.X-Varnish-Store = req.http.X-Varnish-Store;
    } else {
        # remove Varnish fingerprints
        unset resp.http.X-Varnish;
        unset resp.http.Via;
        unset resp.http.X-Powered-By;
        unset resp.http.Server;
        unset resp.http.X-Turpentine-Cache;
        unset resp.http.X-Turpentine-Esi;
        unset resp.http.X-Turpentine-Flush-Events;
        unset resp.http.X-Turpentine-Block;
        unset resp.http.X-Varnish-Session;
        unset resp.http.X-Varnish-Host;
        unset resp.http.X-Varnish-URL;
        # this header indicates the session that originally generated a cached
        # page. it *must* not be sent to a client in production with lax
        # session validation or that session can be hijacked
        unset resp.http.X-Varnish-Set-Cookie;
    }
}

## Backends

backend default {
    .host = "127.0.0.1";
    .port = "8080";
   .first_byte_timeout = 300s;
   .between_bytes_timeout = 300s;
}

backend admin {
    .host = "127.0.0.1";
    .port = "8080";
   .first_byte_timeout = 21600s;
   .between_bytes_timeout = 21600s;
}

## ACLs

#acl crawler_acl {
#    "127.0.0.1";
#}

acl debug_acl {
    "127.0.0.1";
}

## Custom Subroutines

sub generate_session {
    # generate a UUID and add `frontend=$UUID` to the Cookie header, or use SID
    # from SID URL param
    if (req.url ~ ".*[&?]SID=([^&]+).*") {
        set req.http.X-Varnish-Faked-Session = regsub(
            req.url, ".*[&?]SID=([^&]+).*", "frontend=\1");
    } else {
        C{
            char uuid_buf [50];
            generate_uuid(uuid_buf);
            static const struct gethdr_s VGC_HDR_REQ_VARNISH_FAKED_SESSION =
            { HDR_REQ, "\030X-Varnish-Faked-Session:"};
            VRT_SetHdr(ctx,
                &VGC_HDR_REQ_VARNISH_FAKED_SESSION,
                uuid_buf,
                vrt_magic_string_end
            );
        }C
    }
    if (req.http.Cookie) {
        # client sent us cookies, just not a frontend cookie. try not to blow
        # away the extra cookies
        std.collect(req.http.Cookie);
        set req.http.Cookie = req.http.X-Varnish-Faked-Session +
            "; " + req.http.Cookie;
    } else {
        set req.http.Cookie = req.http.X-Varnish-Faked-Session;
    }
}

sub generate_session_expires {
    # sets X-Varnish-Cookie-Expires to now + esi_private_ttl in format:
    #   Tue, 19-Feb-2013 00:14:27 GMT
    # this isn't threadsafe but it shouldn't matter in this case
    C{
        time_t now = time(NULL);
        struct tm now_tm = *gmtime(&now);
        now_tm.tm_sec += 14400;
        mktime(&now_tm);
        char date_buf [50];
        strftime(date_buf, sizeof(date_buf)-1, "%a, %d-%b-%Y %H:%M:%S %Z", &now_tm);
        static const struct gethdr_s VGC_HDR_RESP_COOKIE_EXPIRES =
        { HDR_RESP, "\031X-Varnish-Cookie-Expires:"};
        VRT_SetHdr(ctx,
            &VGC_HDR_RESP_COOKIE_EXPIRES,
            date_buf,
            vrt_magic_string_end
        );
    }C
}

## Varnish Subroutines

sub vcl_recv {
    # this always needs to be done so it's up at the top
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For) {
            set req.http.X-Forwarded-For =
                req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }

    # We only deal with GET and HEAD by default
    # we test this here instead of inside the url base regex section
    # so we can disable caching for the entire site if needed
    if (!true || req.http.Authorization ||
        req.method !~ "^(GET|HEAD)$" ||
        req.http.Cookie ~ "varnish_bypass=1") {
        return (pipe);
    }

    # remove double slashes from the URL, for higher cache hit rate
    set req.url = regsuball(req.url, "(.*)//+(.*)", "\1/\2");

    if (req.http.Accept-Encoding) {
        if (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } else if (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    #if (req.http.User-Agent ~ "(?i)(ads|google|bing|msn|yandex|baidu|ro|career|)bot" ||
    #   req.http.User-Agent ~ "(?i)(baidu|jike|symantec)spider" ||
    #   req.http.User-Agent ~ "(?i)scanner" ||
    #   req.http.User-Agent ~ "(?i)(web)crawler") {
    #   set req.http.X-Normalized-User-Agent = "bot";
    #} else {
    #    set req.http.X-Normalized-User-Agent = "other";
    #}

    # check if the request is for part of magento
    if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
        # set this so Turpentine can see the request passed through Varnish
        set req.http.X-Turpentine-Secret-Handshake = "1";
        # use the special admin backend and pipe if it's for the admin section
        if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?admin") {
            set req.backend_hint = admin;
            return (pipe);
        }
        if (req.http.Cookie ~ "\bcurrency=") {
            set req.http.X-Varnish-Currency = regsub(
                req.http.Cookie, ".*\bcurrency=([^;]*).*", "\1");
        }
        if (req.http.Cookie ~ "\bstore=") {
            set req.http.X-Varnish-Store = regsub(
                req.http.Cookie, ".*\bstore=([^;]*).*", "\1");
        }
        # looks like an ESI request, add some extra vars for further processing
        if (req.url ~ "/turpentine/esi/get(?:Block|FormKey)/") {
            set req.http.X-Varnish-Esi-Method = regsub(
                req.url, ".*/method/(\w+)/.*", "\1");
            set req.http.X-Varnish-Esi-Access = regsub(
                req.url, ".*/access/(\w+)/.*", "\1");

            # throw a forbidden error if debugging is off and a esi block is
            # requested by the user (does not apply to ajax blocks)
            if (req.http.X-Varnish-Esi-Method == "esi" && req.esi_level == 0 &&
                    !(false || client.ip ~ debug_acl)) {
                return (synth(403, "External ESI requests are not allowed"));
            }
        }
        # no frontend cookie was sent to us
        if (req.http.Cookie !~ "frontend=") {
            #if (client.ip ~ crawler_acl ||
            #        req.http.User-Agent ~ "^(?:ApacheBench/.*|.*Googlebot.*|JoeDog/.*Siege.*|magespeedtest\.com|Nexcessnet_Turpentine/.*)$") {
                # it's a crawler, give it a fake cookie
            #    set req.http.Cookie = "frontend=crawler-session";
            #} else {
                # it's a real user, make up a new session for them
                call generate_session;
            #}
        }
        if (true &&
                req.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
            # don't need cookies for static assets
            unset req.http.Cookie;
            unset req.http.X-Varnish-Faked-Session;
            return (hash);
        }
        # this doesn't need a enable_url_excludes because we can be reasonably
        # certain that cron.php at least will always be in it, so it will
        # never be empty
        if (req.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?(?:admin|api|cron\.php|registration/form|oauth|site|scripts)" ||
                # user switched stores. we pipe this instead of passing below because
                # switching stores doesn't redirect (302), just acts like a link to
                # another page (200) so the Set-Cookie header would be removed
                req.url ~ "\?.*__from_store=") {
            return (pipe);
        }
        if (true &&
                req.url ~ "(?:[?&](?:__SID|XDEBUG_PROFILE)(?=[&=]|$))") {
            # TODO: should this be pass or pipe?
            return (pass);
        }
        if (req.url ~ "[?&](utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=") {
            # Strip out Google related parameters
            set req.url = regsuball(req.url, "(?:(\?)?|&)(?:utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=[^&]+", "\1");
            set req.url = regsuball(req.url, "(?:(\?)&|\?$)", "\1");
        }

        # everything else checks out, try and pull from the cache
        return (hash);
    }
    # else it's not part of magento so do default handling (doesn't help
    # things underneath magento but we can't detect that)
}

sub vcl_pipe {
    # since we're not going to do any stuff to the response we pretend the
    # request didn't pass through Varnish
    unset bereq.http.X-Turpentine-Secret-Handshake;
    set bereq.http.Connection = "close";
}

# sub vcl_pass {
#     return (pass);
# }

sub vcl_hash {
    hash_data(req.url);
    if (req.http.Host) {
        hash_data(req.http.Host);
    } else {
        hash_data(server.ip);
    }
    hash_data(req.http.Ssl-Offloaded);
    if (req.http.X-Normalized-User-Agent) {
        hash_data(req.http.X-Normalized-User-Agent);
    }
    if (req.http.Accept-Encoding) {
        # make sure we give back the right encoding
        hash_data(req.http.Accept-Encoding);
    }
    if (req.http.X-Varnish-Store || req.http.X-Varnish-Currency) {
        # make sure data is for the right store and currency based on the *store*
        # and *currency* cookies
        hash_data("s=" + req.http.X-Varnish-Store + "&c=" + req.http.X-Varnish-Currency);
    }

    if (req.http.X-Varnish-Esi-Access == "private" &&
            req.http.Cookie ~ "frontend=") {
        hash_data(regsub(req.http.Cookie, "^.*?frontend=([^;]*);*.*$", "\1"));

    }
    return (lookup);
}

sub vcl_hit {
    # this seems to cause cache object contention issues so removed for now
    # TODO: use obj.hits % something maybe
    # if (obj.hits > 0) {
    #     set obj.ttl = obj.ttl + s;
    # }
}

# sub vcl_miss {
#     return (fetch);
# }

sub vcl_backend_response {
    # set the grace period
    set beresp.grace = 15s;

    # Store the URL in the response object, to be able to do lurker friendly bans later
    set beresp.http.X-Varnish-Host = bereq.http.host;
    set beresp.http.X-Varnish-URL = bereq.url;

    # if it's part of magento...
    if (bereq.url ~ "^(/media/|/skin/|/js/|/)(?:(?:index|litespeed)\.php/)?") {
        # we handle the Vary stuff ourselves for now, we'll want to actually
        # use this eventually for compatibility with downstream proxies
        # TODO: only remove the User-Agent field from this if it exists
        unset beresp.http.Vary;
        # we pretty much always want to do this
        set beresp.do_gzip = true;

        if (beresp.status != 200 && beresp.status != 404) {
            # pass anything that isn't a 200 or 404
            set beresp.ttl = 15s;
            set beresp.uncacheable = true;
            return (deliver);
        } else {
            # if Magento sent us a Set-Cookie header, we'll put it somewhere
            # else for now
            if (beresp.http.Set-Cookie) {
                set beresp.http.X-Varnish-Set-Cookie = beresp.http.Set-Cookie;
                unset beresp.http.Set-Cookie;
            }
            # we'll set our own cache headers if we need them
            unset beresp.http.Cache-Control;
            unset beresp.http.Expires;
            unset beresp.http.Pragma;
            unset beresp.http.Cache;
            unset beresp.http.Age;

            if (beresp.http.X-Turpentine-Esi == "1") {
                set beresp.do_esi = true;
            }
            if (beresp.http.X-Turpentine-Cache == "0") {
                set beresp.ttl = 15s;
                set beresp.uncacheable = true;
                return (deliver);
            } else {
                if (true &&
                        bereq.url ~ ".*\.(?:css|js|jpe?g|png|gif|ico|swf)(?=\?|&|$)") {
                    # it's a static asset
                    set beresp.ttl = 2592000s;
                    set beresp.http.Cache-Control = "max-age=2592000";
                } elseif (bereq.http.X-Varnish-Esi-Method) {
                    # it's a ESI request
                    if (bereq.http.X-Varnish-Esi-Access == "private" &&
                            bereq.http.Cookie ~ "frontend=") {
                        # set this header so we can ban by session from Turpentine
                        set beresp.http.X-Varnish-Session = regsub(bereq.http.Cookie,
                            "^.*?frontend=([^;]*);*.*$", "\1");
                    }
                    if (bereq.http.X-Varnish-Esi-Method == "ajax" &&
                            bereq.http.X-Varnish-Esi-Access == "public") {
                        set beresp.http.Cache-Control = "max-age=" + regsub(
                            bereq.url, ".*/ttl/(\d+)/.*", "\1");
                    }
                    set beresp.ttl = std.duration(
                        regsub(
                            bereq.url, ".*/ttl/(\d+)/.*", "\1s"),
                        300s);
                    if (beresp.ttl == 0s) {
                        # this is probably faster than bothering with 0 ttl
                        # cache objects
                        set beresp.ttl = 15s;
                        set beresp.uncacheable = true;
                        return (deliver);
                    }
                } else {
                    set beresp.ttl = 3600s;
                }
            }
        }
        # we've done what we need to, send to the client
        return (deliver);
    }
    # else it's not part of Magento so use the default Varnish handling
}

nginx 1.6.2 - nginx.conf

user www-data;
worker_processes 32;
worker_rlimit_nofile 100000;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
  worker_connections 4000;
      multi_accept on;
        use epoll;
  }

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  access_log  off;

  sendfile    on;
  tcp_nopush on;

  # Add Support for Real IPs if Varnish is Enabled

  set_real_ip_from 127.0.0.1;
  real_ip_header X-Cluster-Client-Ip;

  server_tokens off;

  types_hash_max_size 2048;
  types_hash_bucket_size 512;

  server_names_hash_bucket_size 64;
  server_names_hash_max_size 512;

  keepalive_timeout 15;
  tcp_nodelay on;

  gzip         on;
  gzip_disable "MSIE [1-6]\.(?!.*SV1)";
  gzip_types   text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

nginx site.conf

server {
  listen                *:8080;

  server_name           xx.co.uk www.xx.co.uk;

  root /var/www/vhosts/magento-capistrano/current;
    index  index.html index.htm index.php;

  access_log            off;

  error_log             /var/log/nginx/xx.co.uk.error.log;    # Base Location
    location / {

        index index.html index.php;    # Display Static File
        try_files $uri $uri/ @handler; # Pass URI to Front Handler
        expires 30d;                   # Cache Assets
    }

    # Denied Locations
    location ^~ /app/                { deny all; }
    location ^~ /includes/           { deny all; }
    location ^~ /lib/                { deny all; }
    location ^~ /media/downloadable/ { deny all; }
    location ^~ /pkginfo/            { deny all; }
    location ^~ /report/config.xml   { deny all; }
    location ^~ /var/                { deny all; }

    # Turn off Logs for Media
    location ~* "^.+\.(jpg|jpeg|gif|css|png|js|ico|pdf|zip|tar|t?gz|mp3|wav|swf|mp4|webm)$" {
        expires    max;
        add_header Cache-Control public;
        access_log off;
    }

   # Hidden Files
    location  /. {
        return 404;
    }

    # Rewrite Internal Requests
    location @handler {
        rewrite / /index.php;
    }

    # Rewrite PHP Requests
    location ~ .php/ {
        rewrite ^(.*.php)/ $1 last;
    }

    # Magento Location
    location ~ \.php$ {
        # Catch 404s
        if (!-e $request_filename) {
            rewrite / /index.php last;
        }

        # Do not cache dynamic content
        expires        off;

        fastcgi_read_timeout 900s;

        fastcgi_pass   unix:/var/run/hhvm/hhvm.sock; # Increase net.core.somaxconn=500 in /etc/sysctl.conf

        # Increase timeout for longer running scripts
        location ~ (/index.php/admin/|/admin/).*$ {
            fastcgi_param PHP_VALUE "max_execution_time = 600";
        }

        fastcgi_param  SCRIPT_FILENAME           $document_root$fastcgi_script_name;
        fastcgi_param  MAGE_RUN_CODE             xx;
        fastcgi_param  MAGE_RUN_TYPE             store;

        include        fastcgi_params;
    }
}

hhvm

HipHop VM 3.4.0-dev~nightly.2014.10.23 (rel)
Compiler: heads/master-0-g0b46339509ebd9612623d52c296fdcc8e6f93dff
Repo schema: 008a6ee43591a5f00ada353a546f193c91f3a4b9
Extension API: 20140829

/etc/sysctl.conf

net.core.netdev_max_backlog=4096

net.core.somaxconn=4096

net.ipv4.tcp_max_syn_backlog=4096

net.netfilter.nf_conntrack_max=131072

net.ipv4.netfilter.ip_conntrack_generic_timeout=120

net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=54000

vm.overcommit_memory=1

/etc/security/limits.conf

* hard memlock 200000
* hard nofile 300000
* soft memlock 82000
* soft nofile 131072

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 966731
max locked memory       (kbytes, -l) 82000
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 966731
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[b3k]

I have similar problem in the past, and the problem was redis session locking, try to disable it - if you are using module Cm_Redis_Session.

[csdougliss]

@b3k No change I'm afraid!

[eth8505]

I think what @b3k mentioned is what happens with the included default config if you have the crawler-session code enabled, as Gatling would most likely always get the frontend=crawler-session code, which does not apply in your case @craigcarnell . Looking at your code, it should work fine though, I don't see any obvious problems...

[nemanjas]

try fix suggested here. https://github.com/nexcess/magento-turpentine/issues/470

[marcobax]

jharrisonau's fix in #470 fixes this problem.

[miguelbalparda]

This seems to be fixed by #470, closing the issue.