srcache_fetch GET with redis2_query and hget

[Roycohen]

Hi There,

I'm trying to use srcache_fetch with Redis2 module via redis2_query and not with the old HTTP REDIS module. I have 2 issues and they might be related.

1. When I'm using redis2 module and not HTTP REDIS in the srcache_fetch directive I never get a good response. Working Example: srcache_fetch GET /redis $key; location = /redis { internal; set $redis_key $args; redis_pass redisbackend_read; }

Non-Working Example: srcache_fetch GET /redis $key; location = /redis { internal; redis2_query get $args redis2_pass redisbackend_read; }

My Upstream backend for both examples is: upstream redisbackend_read { server 127.0.0.1:6379 keepalive 1024; }

I can see that both queries are getting to Redis(I can see in in the Monitor), but when using redis2 it seems like the response is not valid or something because I'm redirected to the backend like the key can't be found.

In the working example there are no errors in the log file but in the non working example with redis2 I can see the following: 2016/02/09 15:25:54 [error] 25052#0: 1 srcachefetch: cache sent invalid status line while sending to client, client: 192.168.56.2, server: , request: "GET /api/adserver/tag?AV_PUBLISHERID=55b88d4a181f465b3e8b4567&AV_CHANNELID=55f030e7181f46b9228b4572 HTTP/1.1", subrequest: "/redis", upstream: "redis2://127.0.0.1:6379", host: "aniview" 2016/02/09 15:25:54 [error] 25052#0: 1 srcachefetch: cache sent truncated response body while sending to client, client: 192.168.56.2, server: , request: "GET /api/adserver/tag?AV_PUBLISHERID=55b88d4a181f465b3e8b4567&AV_CHANNELID=55f030e7181f46b9228b4572 HTTP/1.1", subrequest: "/redis", upstream: "redis2://127.0.0.1:6379", host: "aniview"

Basically I'm trying to use hget against Redis but as I saw that even regular get doesn't work with redis2 module then I think that my issues with hget will be addressed if we can understand what happening here.

NGINX -V output: nginx version: nginx/1.8.0 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --add-module=/usr/src/ngx_http_geoip2_module-1.0 --add-module=/usr/src/nginx-ua-parse-module --with-http_ssl_module --with-http_realip_module --with-http_sub_module --with-http_dav_module --with-http_gunzip_module --with-http_gzip_static_module --with-file-aio --with-ipv6 --with-http_spdy_module --with-cc-opt='-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' --add-module=/usr/src/ngx_http_lower_upper_case --add-module=/usr/src/ngx_http_redis-0.3.7 --add-module=/usr/src/lua-nginx-module-0.9.20 --add-module=/usr/src/redis2-nginx-module-0.12 --add-module=/usr/src/ngx_devel_kit-0.2.19 --add-module=/usr/src/headers-more-nginx-module-0.261 --add-module=/usr/src/set-misc-nginx-module-0.28 --add-module=/usr/src/echo-nginx-module-0.57 --add-module=/usr/src/array-var-nginx-module-0.04 --add-module=/usr/src/strftime-nginx-module --add-module=/usr/src/srcache-nginx-module-0.29

More information from redis for both examples which one works and the other one does not work is: I'm setting the value this way: "set" "zc:k:db6cf5e91987f48d521e85474750b2ca" "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nX-Powered-By: PHP/5.5.32\r\n\r\n"

Any help and investigation will be really appreciated.

[Roycohen]

Attached is debug log when processing the request. It seems like the header is fine and something else is not working.

2016/02/09 15:52:35 [debug] 30682#0: 5 http finalize request: -4, "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" a:1, c:3 2016/02/09 15:52:35 [debug] 30682#0: 5 http request count:3 blk:0 2016/02/09 15:52:35 [debug] 30682#0: 5 http run request: "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http request empty handler 2016/02/09 15:52:35 [debug] 30682#0: 5 http upstream request: "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http upstream process header 2016/02/09 15:52:35 [debug] 30682#0: 5 malloc: 00000000020BF1D0:4096 2016/02/09 15:52:35 [debug] 30682#0: 5 recv: fd:16 126 of 4096 2016/02/09 15:52:35 [debug] 30682#0: 5 srcache_fetch: subrequest returned status 200 2016/02/09 15:52:35 [debug] 30682#0: 5 srcache_fetch decides to send the response in cache 2016/02/09 15:52:35 [debug] 30682#0: 5 http upstream process non buffered downstream 2016/02/09 15:52:35 [debug] 30682#0: 5 http output filter "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http copy filter: "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [error] 30682#0: 5 srcachefetch: cache sent invalid status line while sending to client, client: 192.168.56.2, server: , request: "GET /api/adserver/tag?AV_PUBLISHERID=55b88d4a181f465b3e8b4567&AV_CHANNELID=55f030e7181f46b9228b4572 HTTP/1.1", subrequest: "/redis", upstream: "redis2://127.0.0.1:6379", host: "aniview" 2016/02/09 15:52:35 [debug] 30682#0: 5 http copy filter: 0 "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 finalize http upstream request: 0 2016/02/09 15:52:35 [debug] 30682#0: 5 finalize http redis2 request 2016/02/09 15:52:35 [debug] 30682#0: 5 free keepalive peer 2016/02/09 15:52:35 [debug] 30682#0: 5 free keepalive peer: saving connection 00007F0086ED3298 2016/02/09 15:52:35 [debug] 30682#0: 5 event timer del: 16: 1455025956395 2016/02/09 15:52:35 [debug] 30682#0: 5 free rr peer 2 0 2016/02/09 15:52:35 [debug] 30682#0: 5 http output filter "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http copy filter: "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http copy filter: 0 "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" 2016/02/09 15:52:35 [debug] 30682#0: 5 http finalize request: 0, "/redis?zc:k:db6cf5e91987f48d521e85474750b2ca" a:1, c:2 2016/02/09 15:52:35 [error] 30682#0: 5 srcachefetch: cache sent truncated response body while sending to client, client: 192.168.56.2, server: , request: "GET /api/adserver/tag?AV_PUBLISHERID=55b88d4a181f465b3e8b4567&AV_CHANNELID=55f030e7181f46b9228b4572 HTTP/1.1", subrequest: "/redis", upstream: "redis2://127.0.0.1:6379", host: "aniview" 2016/02/09 15:52:35 [debug] 30682#0: 5 http log handler 2016/02/09 15:52:35 [debug] 30682#0: 5 http wake parent request: "/api/index.php?AV_PUBLISHERID=55b88d4a181f465b3e8b4567&AV_CHANNELID=55f030e7181f46b9228b4572"

[agentzh]

@Roycohen Nope, you need to use the ngx_redis module for the srcache_fetch subrequest because ngx_redis2 always returns the raw Redis responses which are not something expected by the srcache_fetch directive.

For your use case, you can use the ngx_http_lua module's content_by_lua* directives with the lua-resty-redis library in your srcache_fetch subrequest's location. And, use of the OpenResty bundle can simplify your setup.

[Roycohen]

@agentzh, thanks for your quick reply. Is there a way to support ngx_redis2 module or it is too much complicated?

In addition, can you set an example how can I use the ngx_http_lua and the OpenResty bundle, to make an hget request against the Redis.

And one more question, in terms of performance, how much the performance will reduce if I'll use the lua setup instead of the ngx_redis module.

[agentzh]

@Roycohen I don't want to touch the ngx_redis2 module since the lua-resty-redis library is much better.

Regarding an example, it should look like something below (untested):

location = /cache-fetch {
    internal;
    content_by_lua_block {
        local value = assert(ngx.req.get_body_data(), "no value found")
        local key = assert(ngx.req.get_uri_args(), "no key found")
        -- we can log the key to error.log for debugging with this call: print("key is ", key)
        -- ensure you are using exactly the same key as in srcache_store.
        local redis = require "resty.redis"
        local red = redis:new()
        local red = red:set_timeout(1000)  -- ms
        assert(red:connect("127.0.0.1", 6379))
        local data = assert(red:get(key))
        assert(red:set_keepalive(10000, 100))
        if res == ngx.null then
            return ngx.exit(404)
        end
        ngx.print(data)
    }
}

And then use /cache-fetch in your srcache_fetch directive.

I just use the get() in this example. You can use hget() in a similar way. Check out the documentation of the lua-resty-redis library:

https://github.com/openresty/lua-resty-redis#readme

I don't think you can see any measurable performance difference between these two. The syscalls for socket communications are the dominating performance factor here.

[Roycohen]

@agentzh, thank you very very much and have a great day.