Cassandra cluster and Kong Cluster error

[withlin]

Summary

SUMMARY_GOES_HERE

Steps To Reproduce

1. I used cassandra cluster on two machine eg: 192.168.191.131 and 192.168.191.132
2. and then i used kong cluster on two machine. eg:192.168.191.131 and 192.168.191.132
3. but i try down cassandra(192.168.191.132),which i operater that curl localhost:8001.
4. kong is error.

Additional Details & Logs

• Kong version ($ kong version) 0.11.0
• Kong debug-level startup logs ($ kong start --vv) machine 192.168.191.132: root@ubuntu:/home/ubuntu/apache-cassandra-3.11.0/bin# kong start -vv 2017/10/11 20:03:57 [verbose] Kong: 0.11.0 2017/10/11 20:03:57 [debug] ngx_lua: 10008 2017/10/11 20:03:57 [debug] nginx: 1011002 2017/10/11 20:03:57 [debug] Lua: LuaJIT 2.1.0-beta2 2017/10/11 20:03:57 [verbose] reading config file at /etc/kong/kong.conf 2017/10/11 20:03:57 [warn] You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value. 2017/10/11 20:03:57 [debug] admin_access_log = "logs/admin_access.log" 2017/10/11 20:03:57 [debug] admin_error_log = "logs/error.log" 2017/10/11 20:03:57 [debug] admin_http2 = false 2017/10/11 20:03:57 [debug] admin_listen = "0.0.0.0:8001" 2017/10/11 20:03:57 [debug] admin_listen_ssl = "0.0.0.0:8443" 2017/10/11 20:03:57 [debug] admin_ssl = true 2017/10/11 20:03:57 [debug] anonymous_reports = true 2017/10/11 20:03:57 [debug] cassandra_consistency = "QUORUM" 2017/10/11 20:03:57 [debug] cassandra_contact_points = {"192.168.191.131","192.168.191.132"} 2017/10/11 20:03:57 [debug] cassandra_data_centers = {"dc1:2","dc2:3"} 2017/10/11 20:03:57 [debug] cassandra_keyspace = "kong" 2017/10/11 20:03:57 [debug] cassandra_lb_policy = "RoundRobin" 2017/10/11 20:03:57 [debug] cassandra_password = "**" 2017/10/11 20:03:57 [debug] cassandra_port = 9042 2017/10/11 20:03:57 [debug] cassandra_repl_factor = 2 2017/10/11 20:03:57 [debug] cassandra_repl_strategy = "SimpleStrategy" 2017/10/11 20:03:57 [debug] cassandra_schema_consensus_timeout = 10000 2017/10/11 20:03:57 [debug] cassandra_ssl = false 2017/10/11 20:03:57 [debug] cassandra_ssl_verify = false 2017/10/11 20:03:57 [debug] cassandra_timeout = 5000 2017/10/11 20:03:57 [debug] cassandra_username = "cassandra" 2017/10/11 20:03:57 [debug] client_body_buffer_size = "8k" 2017/10/11 20:03:57 [debug] client_max_body_size = "0" 2017/10/11 20:03:57 [debug] client_ssl = false 2017/10/11 20:03:57 [debug] custom_plugins = {} 2017/10/11 20:03:57 [debug] database = "cassandra" 2017/10/11 20:03:57 [debug] db_cache_ttl = 3600 2017/10/11 20:03:57 [debug] db_update_frequency = 5 2017/10/11 20:03:57 [debug] db_update_propagation = 0 2017/10/11 20:03:57 [debug] dns_error_ttl = 1 2017/10/11 20:03:57 [debug] dns_hostsfile = "/etc/hosts" 2017/10/11 20:03:57 [debug] dns_no_sync = false 2017/10/11 20:03:57 [debug] dns_not_found_ttl = 30 2017/10/11 20:03:57 [debug] dns_order = {"LAST","SRV","A","CNAME"} 2017/10/11 20:03:57 [debug] dns_resolver = {"192.168.191.131:8600"} 2017/10/11 20:03:57 [debug] dns_stale_ttl = 4 2017/10/11 20:03:57 [debug] error_default_type = "text/plain" 2017/10/11 20:03:57 [debug] http2 = false 2017/10/11 20:03:57 [debug] latency_tokens = true 2017/10/11 20:03:57 [debug] log_level = "notice" 2017/10/11 20:03:57 [debug] lua_code_cache = "on" 2017/10/11 20:03:57 [debug] lua_package_cpath = "" 2017/10/11 20:03:57 [debug] lua_package_path = "?/init.lua;./kong/?.lua" 2017/10/11 20:03:57 [debug] lua_socket_pool_size = 30 2017/10/11 20:03:57 [debug] lua_ssl_verify_depth = 1 2017/10/11 20:03:57 [debug] mem_cache_size = "128m" 2017/10/11 20:03:57 [debug] nginx_daemon = "on" 2017/10/11 20:03:57 [debug] nginx_optimizations = true 2017/10/11 20:03:57 [debug] nginx_user = "nobody nobody" 2017/10/11 20:03:57 [debug] nginx_worker_processes = "auto" 2017/10/11 20:03:57 [debug] pg_database = "kong" 2017/10/11 20:03:57 [debug] pg_host = "127.0.0.1" 2017/10/11 20:03:57 [debug] pg_password = "**" 2017/10/11 20:03:57 [debug] pg_port = 5432 2017/10/11 20:03:57 [debug] pg_ssl = false 2017/10/11 20:03:57 [debug] pg_ssl_verify = false 2017/10/11 20:03:57 [debug] pg_user = "kong" 2017/10/11 20:03:57 [debug] prefix = "/usr/local/kong/" 2017/10/11 20:03:57 [debug] proxy_access_log = "logs/access.log" 2017/10/11 20:03:57 [debug] proxy_error_log = "logs/error.log" 2017/10/11 20:03:57 [debug] proxy_listen = "0.0.0.0:8000" 2017/10/11 20:03:57 [debug] proxy_listen_ssl = "0.0.0.0:8443" 2017/10/11 20:03:57 [debug] real_ip_header = "X-Real-IP" 2017/10/11 20:03:57 [debug] real_ip_recursive = "off" 2017/10/11 20:03:57 [debug] server_tokens = true 2017/10/11 20:03:57 [debug] ssl = true 2017/10/11 20:03:57 [debug] ssl_cipher_suite = "modern" 2017/10/11 20:03:57 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256" 2017/10/11 20:03:57 [debug] trusted_ips = {} 2017/10/11 20:03:57 [debug] upstream_keepalive = 60 2017/10/11 20:03:57 [verbose] prefix in use: /usr/local/kong 2017/10/11 20:03:57 [verbose] preparing nginx prefix directory at /usr/local/kong 2017/10/11 20:03:58 [verbose] SSL enabled, no custom certificate set: using default certificate 2017/10/11 20:03:58 [verbose] default SSL certificate found at /usr/local/kong/ssl/kong-default.crt 2017/10/11 20:03:58 [verbose] Admin SSL enabled, no custom certificate set: using default certificate 2017/10/11 20:03:58 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt 2017/10/11 20:03:58 [warn] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n" 2017/10/11 20:03:59 [verbose] checking Cassandra schema consensus... 2017/10/11 20:03:59 [verbose] Cassandra schema consensus: reached 2017/10/11 20:03:59 [debug] searching for OpenResty 'nginx' executable 2017/10/11 20:03:59 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.11.2.4' 2017/10/11 20:03:59 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx 2017/10/11 20:03:59 [debug] starting nginx: /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf 2017/10/11 20:04:00 [debug] nginx started 2017/10/11 20:04:00 [info] Kong started

192.168.191.132: root@ubuntu:~# kong start -vv 2017/10/11 20:05:25 [verbose] Kong: 0.11.0 2017/10/11 20:05:25 [debug] ngx_lua: 10008 2017/10/11 20:05:25 [debug] nginx: 1011002 2017/10/11 20:05:25 [debug] Lua: LuaJIT 2.1.0-beta2 2017/10/11 20:05:25 [verbose] reading config file at /etc/kong/kong.conf 2017/10/11 20:05:25 [warn] You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value. 2017/10/11 20:05:25 [debug] admin_access_log = "logs/admin_access.log" 2017/10/11 20:05:25 [debug] admin_error_log = "logs/error.log" 2017/10/11 20:05:25 [debug] admin_http2 = false 2017/10/11 20:05:25 [debug] admin_listen = "0.0.0.0:8001" 2017/10/11 20:05:25 [debug] admin_listen_ssl = "0.0.0.0:8443" 2017/10/11 20:05:25 [debug] admin_ssl = true 2017/10/11 20:05:25 [debug] anonymous_reports = true 2017/10/11 20:05:25 [debug] cassandra_consistency = "QUORUM" 2017/10/11 20:05:25 [debug] cassandra_contact_points = {"192.168.191.131","192.168.191.132"} 2017/10/11 20:05:25 [debug] cassandra_data_centers = {"dc1:2","dc2:3"} 2017/10/11 20:05:25 [debug] cassandra_keyspace = "kong" 2017/10/11 20:05:25 [debug] cassandra_lb_policy = "RoundRobin" 2017/10/11 20:05:25 [debug] cassandra_password = "**" 2017/10/11 20:05:25 [debug] cassandra_port = 9042 2017/10/11 20:05:25 [debug] cassandra_repl_factor = 2 2017/10/11 20:05:25 [debug] cassandra_repl_strategy = "SimpleStrategy" 2017/10/11 20:05:25 [debug] cassandra_schema_consensus_timeout = 10000 2017/10/11 20:05:25 [debug] cassandra_ssl = false 2017/10/11 20:05:25 [debug] cassandra_ssl_verify = false 2017/10/11 20:05:25 [debug] cassandra_timeout = 5000 2017/10/11 20:05:25 [debug] cassandra_username = "cassandra" 2017/10/11 20:05:25 [debug] client_body_buffer_size = "8k" 2017/10/11 20:05:25 [debug] client_max_body_size = "0" 2017/10/11 20:05:25 [debug] client_ssl = false 2017/10/11 20:05:25 [debug] custom_plugins = {} 2017/10/11 20:05:25 [debug] database = "cassandra" 2017/10/11 20:05:25 [debug] db_cache_ttl = 3600 2017/10/11 20:05:25 [debug] db_update_frequency = 5 2017/10/11 20:05:25 [debug] db_update_propagation = 0 2017/10/11 20:05:25 [debug] dns_error_ttl = 1 2017/10/11 20:05:25 [debug] dns_hostsfile = "/etc/hosts" 2017/10/11 20:05:25 [debug] dns_no_sync = false 2017/10/11 20:05:25 [debug] dns_not_found_ttl = 30 2017/10/11 20:05:25 [debug] dns_order = {"LAST","SRV","A","CNAME"} 2017/10/11 20:05:25 [debug] dns_resolver = {"192.168.191.131:8600"} 2017/10/11 20:05:25 [debug] dns_stale_ttl = 4 2017/10/11 20:05:25 [debug] error_default_type = "text/plain" 2017/10/11 20:05:25 [debug] http2 = false 2017/10/11 20:05:25 [debug] latency_tokens = true 2017/10/11 20:05:25 [debug] log_level = "notice" 2017/10/11 20:05:25 [debug] lua_code_cache = "on" 2017/10/11 20:05:25 [debug] lua_package_cpath = "" 2017/10/11 20:05:25 [debug] lua_package_path = "?/init.lua;./kong/?.lua" 2017/10/11 20:05:25 [debug] lua_socket_pool_size = 30 2017/10/11 20:05:25 [debug] lua_ssl_verify_depth = 1 2017/10/11 20:05:25 [debug] mem_cache_size = "128m" 2017/10/11 20:05:25 [debug] nginx_daemon = "on" 2017/10/11 20:05:25 [debug] nginx_optimizations = true 2017/10/11 20:05:25 [debug] nginx_user = "nobody nobody" 2017/10/11 20:05:25 [debug] nginx_worker_processes = "auto" 2017/10/11 20:05:25 [debug] pg_database = "kong" 2017/10/11 20:05:25 [debug] pg_host = "127.0.0.1" 2017/10/11 20:05:25 [debug] pg_port = 5432 2017/10/11 20:05:25 [debug] pg_ssl = false 2017/10/11 20:05:25 [debug] pg_ssl_verify = false 2017/10/11 20:05:25 [debug] pg_user = "kong" 2017/10/11 20:05:25 [debug] prefix = "/usr/local/kong/" 2017/10/11 20:05:25 [debug] proxy_access_log = "logs/access.log" 2017/10/11 20:05:25 [debug] proxy_error_log = "logs/error.log" 2017/10/11 20:05:25 [debug] proxy_listen = "0.0.0.0:8000" 2017/10/11 20:05:25 [debug] proxy_listen_ssl = "0.0.0.0:8443" 2017/10/11 20:05:25 [debug] real_ip_header = "X-Real-IP" 2017/10/11 20:05:25 [debug] real_ip_recursive = "off" 2017/10/11 20:05:25 [debug] server_tokens = true 2017/10/11 20:05:25 [debug] ssl = true 2017/10/11 20:05:25 [debug] ssl_cipher_suite = "modern" 2017/10/11 20:05:25 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256" 2017/10/11 20:05:25 [debug] trusted_ips = {} 2017/10/11 20:05:25 [debug] upstream_keepalive = 60 2017/10/11 20:05:25 [verbose] prefix in use: /usr/local/kong 2017/10/11 20:05:26 [verbose] preparing nginx prefix directory at /usr/local/kong 2017/10/11 20:05:26 [verbose] SSL enabled, no custom certificate set: using default certificate 2017/10/11 20:05:26 [verbose] default SSL certificate found at /usr/local/kong/ssl/kong-default.crt 2017/10/11 20:05:26 [verbose] Admin SSL enabled, no custom certificate set: using default certificate 2017/10/11 20:05:26 [verbose] admin SSL certificate found at /usr/local/kong/ssl/admin-kong-default.crt 2017/10/11 20:05:26 [warn] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n" 2017/10/11 20:05:26 [verbose] checking Cassandra schema consensus... 2017/10/11 20:05:26 [verbose] Cassandra schema consensus: reached 2017/10/11 20:05:26 [debug] searching for OpenResty 'nginx' executable 2017/10/11 20:05:26 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.11.2.4' 2017/10/11 20:05:26 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx 2017/10/11 20:05:26 [debug] starting nginx: /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf 2017/10/11 20:05:27 [debug] nginx started 2017/10/11 20:05:27 [info] Kong started

• Kong error logs (<KONG_PREFIX>/logs/error.log) 2017/10/11 20:11:00 [warn] 3263#0: 1 [lua] socket.lua:152: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua 2017/10/11 20:11:00 [warn] 3262#0: 2 [lua] socket.lua:152: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua 2017/10/11 20:11:30 [warn] 3262#0: 75 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:11:30 [error] 3262#0: 75 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:11:35 [error] 3262#0: 88 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:11:40 [warn] 3262#0: 100 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:11:40 [error] 3262#0: 100 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:11:45 [error] 3262#0: 113 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:11:50 [warn] 3262#0: 125 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:11:50 [error] 3262#0: 125 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:11:55 [error] 3262#0: 138 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:00 [warn] 3262#0: 150 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:12:00 [error] 3262#0: 150 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:05 [error] 3262#0: 163 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:10 [error] 3262#0: 175 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:15 [error] 3262#0: 187 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:20 [warn] 3262#0: 199 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:12:20 [error] 3262#0: 199 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:25 [error] 3262#0: 212 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:30 [error] 3262#0: 224 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:35 [error] 3262#0: 236 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:40 [error] 3262#0: 248 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:45 [error] 3262#0: 260 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:50 [warn] 3262#0: 272 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:12:50 [error] 3262#0: 272 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:12:55 [error] 3262#0: 285 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:00 [error] 3262#0: 297 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:05 [error] 3262#0: 309 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:10 [error] 3262#0: 321 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:15 [error] 3262#0: 333 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:20 [error] 3263#0: 345 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:25 [error] 3262#0: 358 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:30 [warn] 3263#0: 370 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:13:30 [error] 3263#0: 370 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:35 [error] 3262#0: 383 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:40 [warn] 3262#0: 395 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:13:40 [error] 3262#0: 395 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:45 [error] 3263#0: 408 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:50 [error] 3262#0: 420 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:13:55 [error] 3263#0: 432 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:00 [error] 3262#0: 443 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:05 [error] 3263#0: 455 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:10 [error] 3263#0: 467 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:15 [error] 3262#0: 479 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:20 [error] 3263#0: 491 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:25 [error] 3263#0: 503 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:30 [warn] 3262#0: 515 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:14:30 [error] 3262#0: 515 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:35 [error] 3263#0: 528 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:40 [error] 3262#0: 540 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:45 [error] 3263#0: 552 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:50 [error] 3262#0: 564 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:14:55 [error] 3263#0: 576 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:00 [error] 3262#0: 588 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:05 [error] 3263#0: 600 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:10 [error] 3262#0: 612 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:15 [error] 3263#0: 624 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:20 [error] 3262#0: 636 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:25 [error] 3263#0: 648 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:30 [warn] 3262#0: 660 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:15:30 [error] 3262#0: 660 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:35 [error] 3263#0: 673 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:40 [error] 3263#0: 685 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:45 [error] 3263#0: 697 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:50 [error] 3263#0: 709 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:15:55 [error] 3263#0: 722 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:00 [error] 3262#0: 733 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:05 [error] 3263#0: 745 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:10 [error] 3262#0: 757 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:15 [error] 3262#0: 769 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:20 [error] 3263#0: 781 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:25 [error] 3263#0: 793 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:30 [warn] 3262#0: 805 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:16:30 [error] 3262#0: 805 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:35 [error] 3263#0: 818 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:40 [error] 3262#0: 830 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:45 [error] 3263#0: 842 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:50 [error] 3262#0: 854 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:16:55 [error] 3263#0: 866 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:00 [error] 3263#0: 878 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:05 [error] 3262#0: 890 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:10 [error] 3263#0: 902 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:15 [error] 3262#0: 914 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:20 [error] 3263#0: 926 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:25 [error] 3262#0: 938 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:30 [warn] 3262#0: 950 [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.191.132 DOWN, context: ngx.timer 2017/10/11 20:17:30 [error] 3262#0: 950 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:35 [error] 3263#0: 963 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:40 [error] 3262#0: 975 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:45 [error] 3263#0: 987 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:50 [error] 3263#0: 999 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer 2017/10/11 20:17:55 [error] 3262#0: 1011 [lua] cluster_events.lua:369: [cluster_events] failed to poll: failed to retrieve events from DB: [Unavailable exception] Cannot achieve consistency level ONE, context: ngx.timer
• Kong configuration (registered APIs/Plugins & configuration file)
• Operating System ubuntu16.04.3

[withlin]

please help me.

[thibaultcha]

This sounds like a mis-configured Cassandra cluster and/or keyspace. We don't have enough information here, nor access to your Cassandra cluster to help you debug this issue further, but you can find plenty of resources online to help you with Cassandra, including troubleshooting guides for the error you encountered.

[yousafsyed]

Same issue here using the konghq kubernetes deployment https://github.com/Kong/kong-dist-kubernetes by following all the instructions as is

[code2design]

@kikito I'm getting the same error while using their official Kong 0.14.x docker image to deploy it on openshift origin. Is there a way to create and change the user to achieve the same behavior?

[thibaultcha]

@yousafsyed @code2design This issue usually relates to a misconfiguration of your Cassandra cluster. See this message for a starting point to understand the issue and fix it: https://github.com/Kong/kong/issues/3596#issuecomment-403159132

[yousafsyed-bg]

@thibaultcha you are probably right but Cassandra configurations are also provided with the kong's kubernetes manifesto. https://github.com/Kong/kong-dist-kubernetes/blob/master/kong_cassandra.yaml So if the Cassandra is misconfigured in the repo than we should fix that, I don't see anything particularly wrong here in the manifesto for now but I am also not an expert at Cassandra.