varnish / varnish-modules

Collection of Varnish Cache modules (vmods) by Varnish Software
Other
182 stars 86 forks source link

vmod_softpurge: frequent cache-worker crashes "Assert error in hsh_rush1()..." #60

Closed cnMiJohn closed 6 years ago

cnMiJohn commented 7 years ago

Hi there,

we are experiencing irregular child crashes since using the softpurge module.

Panic at: Thu, 23 Feb 2017 12:00:07 GMT
"Assert error in hsh_rush1(), cache/cache_hash.c line 531:
  Condition(max > 0) not true.
thread = (cache-worker)
version = varnish-5.0.0 revision 99d036f
ident = Linux,4.4.35-33.55.amzn1.x86_64,x86_64,-junix,-sfile,-smalloc,-hcritbit,epoll
Backtrace:
  0x4342f3: pan_ic+0x163
  0x428173: hsh_rush1+0x1c3
  0x42985c: HSH_DerefObjCore+0xec
  0x7fb5585fbc70: libvmod_softpurge.so(vmod_softpurge+0x210) [0x7fb5585fbc70]
  0x7fb69c5a6326: vgc.so(VGC_function_vcl_hit+0x136) [0x7fb69c5a6326]
  0x441b25: vcl_call_method+0x1f5
  0x4426ca: VCL_hit_method+0x5a
  0x4382ca: cnt_lookup+0x16a
  0x438bd1: CNT_Request+0x411
  0x4545af: HTTP1_Session+0x4ef
req = 0x7fb40973a020 {
  vxid = 43483677, transport = HTTP/1 {
    state = HTTP1::Proc
  }
  step = R_STP_LOOKUP,
  req_body = R_BODY_NONE,
  restarts = 0, esi_level = 0,
  sp = 0x7fb40c902620 {
    fd = 36, vxid = 43483676,
    t_open = 1487851206.799406,
    t_idle = 1487851206.799406,
    transport = HTTP/1 {
      state = HTTP1::Proc
    }
client = 127.0.0.1 26270,
  },
  worker = 0x7fb6a502cc10 {
    stack = {0x7fb6a502d000 -> 0x7fb6a5021000},
    ws = 0x7fb6a502cdf8 {
      id = \"wrk\",
      {s, f, r, e} = {0x7fb6a502c3d0, 0x7fb6a502c3d0, (nil), +2040},
    },
    VCL::method = inside HIT,
    VCL::return = 0x0,
    VCL::methods = {HIT},
  },
  ws = 0x7fb40973a1f8 {
    id = \"req\",
    {s, f, r, e} = {0x7fb40973bff8, +312, +57344, +57344},
  },
  http_conn = 0x7fb40973a128 {
    fd = 36,
    doclose = NULL,
    ws = 0x7fb40973a1f8,
    {rxbuf_b, rxbuf_e} = {0x7fb40973bff8, 0x7fb40973c05a},
    {pipeline_b, pipeline_e} = {(nil), (nil)},
    content_length = -1,
    body_status = none,
    first_byte_timeout = 0.000000,
    between_bytes_timeout = 0.000000,
  },
  http[req] = 0x7fb40973a290 {
    ws[req] = 0x7fb40973a1f8,
    hdrs {
      \"PURGE\",
      \"###URL###\",
      \"HTTP/1.1\",
      \"Host: ###HOST###\",
      \"Connection: Close\",
      \"X-Environment: ###ENV###\",
      \"X-Internal: 0\",
      \"X-Forwarded-For: 127.0.0.1\",
      \"X-Domain: ###DOMAIN###\",
      \"X-UA-Device: ###DEVICE###\",
    },
  },
  vcl = {
    name = \"boot\"
    busy = 178
    discard = 0,
    state = auto,
    temp = warm,
    conf = {
      srcname = {
        ...
      },
    },
  },
  objcore[REQ] = 0x7fb40cbe6880 {
    refcnt = 4,
    flags = {},
    exp_flags = {refd},
    boc = 0x7fb40cb28e00 {
      refcnt = 3,
      state = stream,
      vary = 0x7fb40ea081f0,
      stevedore_priv = (nil),
    },
    exp = {1487851206.818771, 0.000515, 86400.000000, 0.000000}
    objhead = 0x7fb40e78b5f0,
    stevedore = 0x7fb6a48eb180 (file s0) {
      Simple = 0x7fb62fd7e000
      Obj 0x7fb409ce9e28 {priv=0x7fb409ce9e20, ptr=0x7fb62fd7e000, len=669, space=4096}
      LEN = 0x0...0
      VXID = 0x01438d61
      FLAGS = 0x0a
      GZIPBITS = 0x00000000000000500000000000000050000000000001303b000000000000ec68
      LASTMODIFIED = 0x41d62bb3afc00000
      VARY = {len=0, ptr=0x7fb62fd7e078}
      HEADERS = {len=0, ptr=0x7fb62fd7e0bd}
      Body 0x7fb40eb10c48 {priv=0x7fb40eb10c40, ptr=0x7fb62fd84000, len=9744, space=12288}
    },
  },
  flags = {
  },
},
"

Is that an issue of the softpurge module or varnish itself?

nigoroll commented 6 years ago

ftr, vmod_softpurge is now replaced with vmod_purge in varnish-cache (as of 5.2)