Stream based load balancing not working

[sachinkale]

Hello,

I configured stream based configuration and loaded upstream hosts via consul.

The load balancing fails as nginx is not able to get upstream hosts, it gives me this error -

2018/01/25 10:53:04 [info] 18768#0: *26286 client 220.225.68.105:1312 connected to 0.0.0.0:32004
2018/01/25 10:53:04 [error] 18768#0: *26286 stream [lua] balance.lua:32: select peer failed, unknown skey 32004 while connecting to upstream, client: 220.225.68.105, server: 0.0.0.0:32004, bytes from/to client:0/0, bytes from/to upstream:0/0
2018/01/25 10:53:04 [crit] 18768#0: *26286 connect() to 0.0.0.2:80 failed (22: Invalid argument) while connecting to upstream, client: 220.225.68.105, server: 0.0.0.0:32004, upstream: "0.0.0.2:80", bytes from/to client:0/0, bytes from/to upstream:0/0

Here is the upstream that is configured [ec2-user@ip-10-243-17-41 nginx]$ curl http://127.0.0.1:1995/upstreams {"mybackend":{"cluster":[{"servers":[{"host":"10.243.17.26","weight":1,"fail_timeout":30,"port":32004,"max_fails":6}]}],"enable":true},"logger":{"enable":false,"node_type":"slardar_access","cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"node_host":"127.0.0.1","timeout":2,"config":{"flush_limit":4096,"ups_name":"logger","drop_limit":1048576,"pool_size":10,"retry_interval":100,"max_retry_times":3}},"store":{"enable":false,"config_cache_enable":true,"config_positive_ttl":10,"cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"type":"consul","config_negative_ttl":5,"timeout":5,"config_key_prefix":"config\/slardar\/"}}[ec2-user@ip-10-243-17-41 nginx]$

Let me know how to resolve this, feel free to point out source files which I can modify .

-Sachin

[yejingx]

Can you show me your nginx.conf?

[sachinkale]

Nginx.conf (did not change, as is from the repo)

---

user nobody; worker_processes auto;

error_log logs/error.log;

error_log logs/error.log notice;

error_log logs/error.log info;

pid logs/nginx.pid;

events { worker_connections 1024; }

stream { lua_check_client_abort on; lua_socket_log_errors off;

lua_shared_dict stream_state  10m;
lua_shared_dict stream_locks  10m;
lua_shared_dict stream_cache  10m;
lua_shared_dict stream_mutex  1m;
lua_shared_dict stream_config 10m;
lua_shared_dict stream_load   10m;

lua_package_path "$prefix/app/lib/?.lua;$prefix/app/etc/?.lua;$prefix/app/src/?.lua;$prefix/app/load/?.lua;$prefix/../luajit/share/lua/5.1/?.lua;;";
lua_package_cpath "$prefix/app/lib/?.so;$prefix/../luajit/lib/lua/5.1/?.so;;";

proxy_next_upstream_tries 3;

init_by_lua_file app/src/init.lua;
init_worker_by_lua_file app/src/init_worker.lua;
include stream/stream.conf;

}

http { include mime.types; default_type application/octet-stream;

access_log  off;

log_format  main '$remote_addr - $remote_user [$time_local] '
'"$request_method $scheme://$host$request_uri $server_protocol" '
'$status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" '
'"$upstream_addr" $upstream_status $upstream_response_time $request_time';

uninitialized_variable_warn off;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  15;

send_timeout 60;
proxy_read_timeout 75;
proxy_send_timeout 60;
proxy_connect_timeout 3;
proxy_max_temp_file_size 1m;

client_header_buffer_size 4k;
large_client_header_buffers 8 16k;
client_max_body_size 1024m;

port_in_redirect off;

open_log_file_cache max=2048 inactive=60s min_uses=2 valid=15m;

lua_shared_dict state 50m;
lua_shared_dict locks 10m;
lua_shared_dict mutex 1m;
lua_shared_dict cache 50m;
lua_shared_dict config 50m;
lua_shared_dict load   50m;

lua_max_running_timers 512;
lua_max_pending_timers 1024;

lua_package_path "$prefix/app/lib/?.lua;$prefix/app/etc/?.lua;$prefix/app/src/?.lua;$prefix/../luajit/share/lua/5.1/?.lua;;";
lua_package_cpath "$prefix/app/lib/?.so;$prefix/../luajit/lib/lua/5.1/?.so;;";

lua_check_client_abort on;
lua_socket_log_errors off;
lua_http10_buffering off;

# lua_code_cache off;

init_by_lua_file app/src/init.lua;
init_worker_by_lua_file app/src/init_worker.lua;

include slardar/*.conf;

}

---

In conf/stream/stream.conf I added my upstream

---

upstream mybackend { server 0.0.0.2:80; balancer_by_lua_file app/src/balance.lua;

}

server { listen 32004; proxy_pass mybackend; }

server { listen 3141; preread_by_lua_file app/src/stream/preread.lua; content_by_lua_file app/src/stream/content.lua; log_by_lua_file app/src/log.lua; }

server { listen 1895;

content_by_lua_file app/src/stream/status.lua;

}

---

[huangnauh]

[ec2-user@ip-10-243-17-41 nginx]$ curl http://127.0.0.1:1995/upstreams

echo -ne 'get upstream info\n' | nc 127.0.0.1 1895 in stream subsystem

https://github.com/upyun/slardar/blob/master/nginx/app/src/modules/protocol.lua#L104

2018/01/25 10:53:04 [info] 18768#0: *26286 client 220.225.68.105:1312 connected to 0.0.0.0:32004

Depending on your situation, nginx need listen for stream connections on port 32004.

[huangnauh]

select peer failed, unknown skey 32004

https://github.com/upyun/slardar/blob/master/nginx/app/src/balance.lua#L15

stream subsystem will use server_port as upstream skey in checkups

[sachinkale]

Here is the o/p of stream subsystem [ec2-user@ip-10-243-17-41 nginx]$ echo -ne 'get upstream info\n' | nc 127.0.0.1 1895 �{"logger":{"enable":false,"node_type":"slardar_access","cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"node_host":"127.0.0.1","timeout":2,"config":{"flush_limit":4096,"ups_name":"logger","drop_limit":1048576,"pool_size":10,"retry_interval":100,"max_retry_times":3}},"store":{"enable":false,"config_cache_enable":true,"config_positive_ttl":10,"cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"type":"consul","config_negative_ttl":5,"timeout":5,"config_key_prefix":"config\/slardar\/"}}

Does it means that the upstream configured are only for http?

How do we configure upstreams via consul or URL for stream subsystem?

[huangnauh]

How do we configure upstreams via consul or URL for stream subsystem?

key_prefix .. "upstreams_stream/" for the stream subsystem, just like key_prefix .. "upstreams/" in the http subsystem

https://github.com/upyun/slardar/blob/master/nginx/app/lib/resty/store/config.lua#L65-L70

[sachinkale]

Thanks for your kind inputs, I think I made progress, I was able to configure upstreams in stream subsystem via "upstream_stream" key prefix in consul

---

[ec2-user@ip-10-243-17-41 nginx]$ ~/consul kv get config/slardar/upstreams_stream/mybackend
{"enable": true, "servers": [{"host": "10.243.17.26","port": 32004,"weight": 1,"max_fails": 6,"fail_timeout": 30}]}

[ec2-user@ip-10-243-17-41 nginx]$ echo -ne 'get upstream info\n' | nc 127.0.0.1 1895
�{"mybackend":{"cluster":[{"servers":[{"host":"10.243.17.26","weight":1,"fail_timeout":30,"port":32004,"max_fails":6}]}],"enable":true},"logger":{"enable":false,"node_type":"slardar_access","cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"node_host":"127.0.0.1","timeout":2,"config":{"flush_limit":4096,"ups_name":"logger","drop_limit":1048576,"pool_size":10,"retry_interval":100,"max_retry_times":3}},"store":{"enable":false,"config_cache_enable":true,"config_positive_ttl":10,"cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"type":"consul","config_negative_ttl":5,"timeout":5,"config_key_prefix":"config\/slardar\/"}}[ec2-user@ip-10-243-17-41 nginx]$

---

However, the original problem remains unsolved, I am still not getting connected to port 32004 through Nginx, it gives the same error:

---

2018/01/26 02:46:43 [info] 20844#0: *1403 client 10.243.17.27:40460 connected to 0.0.0.0:32004
2018/01/26 02:46:43 [error] 20844#0: *1403 stream [lua] balance.lua:32: select peer failed, unknown skey 32004 while connecting to upstream, client: 10.243.17.27, server: 0.0.0.0:32004, bytes from/to client:0/0, bytes from/to upstream:0/0
2018/01/26 02:46:43 [crit] 20844#0: *1403 connect() to 0.0.0.2:80 failed (22: Invalid argument) while connecting to upstream, client: 10.243.17.27, server: 0.0.0.0:32004, upstream: "0.0.0.2:80", bytes from/to client:0/0, bytes from/to upstream:0/0

---

[huangnauh]

skey error: https://github.com/upyun/slardar/issues/41#issuecomment-360500630

Depending on your situation, port 32004 will get upstreams from config/slardar/upstreams_stream/32004

[sachinkale]

Setting config/slardar/upstreams_stream/32004 worked perfectly, Thanks.

There are two upstreams seen in status (mybackend, 32004):

{"mybackend":{"cluster":[{"servers":[{"host":"10.243.17.26","weight":1,"fail_timeout":30,"port":32004,"max_fails":6}]}],"enable":true},"store":{"enable":false,"config_cache_enable":true,"config_positive_ttl":10,"cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"type":"consul","config_negative_ttl":5,"timeout":5,"config_key_prefix":"config\/slardar\/"},"logger":{"enable":false,"node_type":"slardar_access","cluster":[{"servers":[{"host":"127.0.0.1","port":8500}]}],"node_host":"127.0.0.1","timeout":2,"config":{"flush_limit":4096,"ups_name":"logger","drop_limit":1048576,"pool_size":10,"retry_interval":100,"max_retry_times":3}},"32004":{"cluster":[{"servers":[{"host":"10.243.17.26","weight":1,"fail_timeout":30,"port":32004,"max_fails":6}]}],"enable":true}}

Is this a bug?

Shouldn't "skey" be the upstream name instead of port number?

How should I add/remove servers?

This did not worked

curl 127.0.0.1:1995/upstream_stream/32004 -d '{"servers": [{"host": "10.243.17.26","port": 32004,"weight": 1,"max_fails": 6,"fail_timeout": 30}, {"host": "10.243.17.27","port": 32004,"weight": 1,"max_fails": 6,"fail_timeout": 30}]}'

Also updating consul key also is not working. Please guide.

Thanks for your help!

-Sachin

[huangnauh]

updating consul key also is not working.

use the tcp protocol to manipulate upstream in stream subsystem

defined by https://github.com/upyun/slardar/blob/master/nginx/app/src/modules/protocol.lua

http://huangnauh.github.io/2017OpenRestyMeetup_files/protocol.png

for example

PUT $ echo -ne 'PUT upstream 3131\n\x00\0x00\0x00\0x3d{"cluster":[{"servers":[{"host":"127.0.0.1", "port":3306}]}]}' | nc 127.0.0.1 1895 | xxd

GET $ echo -ne 'get upstream info\n' | nc 127.0.0.1 1895

stream docs still work in progress...

Sorry for the late reply