shrpx not forwarding on requests to backend

[thehaven]

Hi,

I have both a working setup and a broken setup on two separate machines and using identical configs and identical versions of spdylay (1.1.0). The working setup is Gentoo Unstable, the non-working is a new machine build on Gentoo stable so there are library differences. The primary difference between the two systems however is that the "stable" build is using ZFS (zfsonlinux) as its primary file system.

My setup sends packets requesting spdy/3 from HAProxy -> shrpx -> Nginx.

On the non-working setup I can see that requests are hitting the shrpx server. However no traffic is forwarded on. Here are some logs:

Generating the request via spdycat:

[  0.024] NPN select next protocol: the remote server offers:
          * spdy/3
          NPN selected the protocol: spdy/3
[  0.026] Handshake complete
[  0.027] send SYN_STREAM frame <version=3, flags=1, length=238>
          (stream_id=1, assoc_stream_id=0, pri=3)
          :host: nostromo.thehavennet.org.uk
          :method: GET
          :path: /test.php
          :scheme: https
          :version: HTTP/1.1
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: spdylay/1.1.0
Fatal: EOF
reqnum=1, completed=0
Some requests were not processed. total=1, processed=0

Monitoring the output from shrpx:

[INFO] [LISTEN:0x23c4830] Accepted connection. fd=27
       (shrpx_listen_handler.cc:97)
[INFO] [THREAD_RECV:0x7f7140001090] WorkerEvent: client_fd=27, addrlen=16
       (shrpx_thread_event_receiver.cc:56)
127.0.0.1 [Thu Nov 28 06:16:56 2013] ACCEPT
[INFO] [THREAD_RECV:0x7f7140001090] CLIENT_HANDLER:0x7f7140002d80 created
       (shrpx_thread_event_receiver.cc:68)
[INFO] [CLIENT_HANDLER:0x7f7140002d80] Network error: Success
       (shrpx_client_handler.cc:86)
[INFO] [CLIENT_HANDLER:0x7f7140002d80] Deleting
       (shrpx_client_handler.cc:157)
[INFO] [CLIENT_HANDLER:0x7f7140002d80] Deleted
       (shrpx_client_handler.cc:176)

"Network error: Success" is confusing me a bit - does that mean there is a network error?

Monitoring the outgoing port on shrpx via tcpdump I can confirm that no traffic is sent.

I've also tried running shrpx on this machine using both master and v1.2.1 with no success. Given ZFS's as the primary difference between the two machines I'm wondering whether that is the contributing factor here. Here's a snippet from strace that suggests an epoll issue perhaps? (out of my depth a bit here unfortunately):

epoll_ctl(3, EPOLL_CTL_DEL, 7, {...})   = 0
epoll_wait(3, {?} 0xc04b00, 32, -1)     = 1
accept(6, {sa_family=AF_INET, sin_port=htons(46819), sin_addr=inet_addr("127.0.0.1")}, [16]) = 27
fcntl(27, F_GETFL)                      = 0x2 (flags O_RDWR)
fcntl(27, F_SETFL, O_RDWR|O_NONBLOCK)   = 0
write(2, "[\33[1;32mINFO\33[0m] [LISTEN:0xc048"..., 110[INFO] [LISTEN:0xc04830] Accepted connection. fd=27
       (shrpx_listen_handler.cc:97)
) = 110
epoll_ctl(3, EPOLL_CTL_ADD, 12, {...})  = 0
accept(6, 0x7fff5056bcf0, [128])        = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(3, {?} 0xc04b00, 32, -1)     = 1

I'm not really sure what to try next and would appreciate any assistance.

Many thanks.

[tatsuhiro-t]

I don't know much about zfs, but it is just a file system, and not likely to affect networking stack. If libevent polls file system changes on zfs using epoll, then it may fail, but we apparently don't use file change event.

The strace log looks normal to me. The "network error: success" is confusing, but it usually means that the underlying socket was closed gracefully.

With your setup, which server does the spdycat perform SSL/TLS handshake? HAproxy or shrpx? If HAproxy forwards decrypted stream to shrpx, make sure that shrpx is configured with --frontend-spdy-no-tls and SPDY version is also specified with --frontend-spdy-proto. It would be advisable to check again the configuration, because people are likely missing to specify correct config file path, etc.

[thehaven]

Thanks @tatsuhiro-t,

It was indeed the configuration parameters I was passing. I'd accidentally over-ridden my old settings on the new machine and switched from:

SHRPX_CONF="-D -k --add-x-forwarded-for --no-via --frontend-spdy-no-tls --honor-cipher-order --conf /etc/shrpx.conf"

To: SHRPX_CONF="-D -k --add-x-forwarded-for --conf /etc/shrpx.conf"

Fixing that fixed the issue.

I really should have spotted that. Thanks for pointing me in the right direction.