search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/
Apache License 2.0
39.38k stars 4.83k forks source link

Kong 1.0 rc2 does not seem to forward http logs with HTTP-log plugin #3970

Closed rahimimo closed 6 years ago

rahimimo commented 6 years ago

Summary

Kong 1.0 rc2

Steps To Reproduce

  1. With the exactly the same environment with kong 1 and kong 14 with the correct migrations- Kong 1 rc 2 does not forward the logs to my upstream server which is logstash via http-log plugin. I can see the plugin in cassandra and through plugins rest endpoint. However it does not seem to be operating.

Additional Details & Logs

The output of global plugins is (in 1.0 rc2): {"next":null,"data":[{"created_at":1542013696,"config":{"method":"POST","http_endpoint":"http:\/\/192.168.228.108:5043\/","retry_count":10,"timeout":10000,"queue_size":1,"keepalive":60000,"content_type":"application\/json","flush_timeout":2},"api":null,"service":null,"enabled":true,"id":"f29daff3-99ea-469d-8d90-0a45f403f439","consumer":null,"route":null,"name":"http-log"}]} But in kong 14 it was {"total":1,"data":[{"created_at":1541756790400,"config":{"http_endpoint":"http:\/\/192.168.228.108:5043\/","timeout":10000,"method":"POST","keepalive":60000,"content_type":"application\/json"},"id":"a67c3b22-251b-43ec-9d77-1618fc15f055","name":"http-log","enabled":true}]}

RobSerafini commented 6 years ago

rahimimo

Can you also provide the following additional details?

Regards, Rob

rahimimo commented 6 years ago

Hi @RobSerafini Kong Start -vv

2018/11/14 08:29:01 [verbose] Kong: 1.0.0rc2
2018/11/14 08:29:01 [debug] ngx_lua: 10013
2018/11/14 08:29:01 [debug] nginx: 1013006
2018/11/14 08:29:01 [debug] Lua: LuaJIT 2.1.0-beta3
2018/11/14 08:29:01 [verbose] no config file found at /etc/kong/kong.conf
2018/11/14 08:29:01 [verbose] no config file found at /etc/kong.conf
2018/11/14 08:29:01 [verbose] no config file, skip loading
2018/11/14 08:29:01 [debug] reading environment variables
2018/11/14 08:29:01 [debug] KONG_ADMIN_LISTEN ENV found with "0.0.0.0:8001"
2018/11/14 08:29:01 [debug] KONG_DB_UPDATE_PROPAGATION ENV found with "1"
2018/11/14 08:29:01 [debug] KONG_DATABASE ENV found with "cassandra"
2018/11/14 08:29:01 [debug] KONG_CASSANDRA_CONSISTENCY ENV found with "ONE"
2018/11/14 08:29:01 [debug] KONG_CASSANDRA_REPL_STRATEGY ENV found with "SimpleStrategy"
2018/11/14 08:29:01 [debug] KONG_CASSANDRA_CONTACT_POINTS ENV found with "cassandra"
2018/11/14 08:29:01 [debug] KONG_LOG_LEVEL ENV found with "debug"
2018/11/14 08:29:01 [debug] KONG_CASSANDRA_REPL_FACTOR ENV found with "3"
2018/11/14 08:29:01 [debug] admin_access_log = "logs/admin_access.log"
2018/11/14 08:29:01 [debug] admin_error_log = "logs/error.log"
2018/11/14 08:29:01 [debug] admin_listen = {"0.0.0.0:8001"}
2018/11/14 08:29:01 [debug] anonymous_reports = true
2018/11/14 08:29:01 [debug] cassandra_consistency = "ONE"
2018/11/14 08:29:01 [debug] cassandra_contact_points = {"cassandra"}
2018/11/14 08:29:01 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2018/11/14 08:29:01 [debug] cassandra_keyspace = "kong"
2018/11/14 08:29:01 [debug] cassandra_lb_policy = "RoundRobin"
2018/11/14 08:29:01 [debug] cassandra_port = 9042
2018/11/14 08:29:01 [debug] cassandra_repl_factor = 3
2018/11/14 08:29:01 [debug] cassandra_repl_strategy = "SimpleStrategy"
2018/11/14 08:29:01 [debug] cassandra_schema_consensus_timeout = 10000
2018/11/14 08:29:01 [debug] cassandra_ssl = false
2018/11/14 08:29:01 [debug] cassandra_ssl_verify = false
2018/11/14 08:29:01 [debug] cassandra_timeout = 5000
2018/11/14 08:29:01 [debug] cassandra_username = "kong"
2018/11/14 08:29:01 [debug] client_body_buffer_size = "8k"
2018/11/14 08:29:01 [debug] client_max_body_size = "0"
2018/11/14 08:29:01 [debug] client_ssl = false
2018/11/14 08:29:01 [debug] custom_plugins = {}
2018/11/14 08:29:01 [debug] database = "cassandra"
2018/11/14 08:29:01 [debug] db_cache_ttl = 0
2018/11/14 08:29:01 [debug] db_resurrect_ttl = 30
2018/11/14 08:29:01 [debug] db_update_frequency = 5
2018/11/14 08:29:01 [debug] db_update_propagation = 1
2018/11/14 08:29:01 [debug] dns_error_ttl = 1
2018/11/14 08:29:01 [debug] dns_hostsfile = "/etc/hosts"
2018/11/14 08:29:01 [debug] dns_no_sync = false
2018/11/14 08:29:01 [debug] dns_not_found_ttl = 30
2018/11/14 08:29:01 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2018/11/14 08:29:01 [debug] dns_resolver = {}
2018/11/14 08:29:01 [debug] dns_stale_ttl = 4
2018/11/14 08:29:01 [debug] error_default_type = "text/plain"
2018/11/14 08:29:01 [debug] headers = {"server_tokens","latency_tokens"}
2018/11/14 08:29:01 [debug] log_level = "debug"
2018/11/14 08:29:01 [debug] lua_package_cpath = ""
2018/11/14 08:29:01 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2018/11/14 08:29:01 [debug] lua_socket_pool_size = 30
2018/11/14 08:29:01 [debug] lua_ssl_verify_depth = 1
2018/11/14 08:29:01 [debug] mem_cache_size = "128m"
2018/11/14 08:29:01 [debug] nginx_admin_directives = {}
2018/11/14 08:29:01 [debug] nginx_daemon = "on"
2018/11/14 08:29:01 [debug] nginx_http_directives = {}
2018/11/14 08:29:01 [debug] nginx_optimizations = true
2018/11/14 08:29:01 [debug] nginx_proxy_directives = {}
2018/11/14 08:29:01 [debug] nginx_user = "nobody nobody"
2018/11/14 08:29:01 [debug] nginx_worker_processes = "auto"
2018/11/14 08:29:01 [debug] origins = {}
2018/11/14 08:29:01 [debug] pg_database = "kong"
2018/11/14 08:29:01 [debug] pg_host = "127.0.0.1"
2018/11/14 08:29:01 [debug] pg_port = 5432
2018/11/14 08:29:01 [debug] pg_ssl = false
2018/11/14 08:29:01 [debug] pg_ssl_verify = false
2018/11/14 08:29:01 [debug] pg_timeout = 5000
2018/11/14 08:29:01 [debug] pg_user = "kong"
2018/11/14 08:29:01 [debug] plugins = {"bundled"}
2018/11/14 08:29:01 [debug] prefix = "/usr/local/kong/"
2018/11/14 08:29:01 [debug] proxy_access_log = "logs/access.log"
2018/11/14 08:29:01 [debug] proxy_error_log = "logs/error.log"
2018/11/14 08:29:01 [debug] proxy_listen = {"0.0.0.0:8000","0.0.0.0:8443 ssl"}
2018/11/14 08:29:01 [debug] real_ip_header = "X-Real-IP"
2018/11/14 08:29:01 [debug] real_ip_recursive = "off"
2018/11/14 08:29:01 [debug] ssl_cipher_suite = "modern"
2018/11/14 08:29:01 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
2018/11/14 08:29:01 [debug] trusted_ips = {}
2018/11/14 08:29:01 [debug] upstream_keepalive = 60
2018/11/14 08:29:01 [verbose] prefix in use: /usr/local/kong
2018/11/14 08:29:01 [debug] loading subsystems migrations...
2018/11/14 08:29:01 [verbose] retrieving keyspace schema state...
2018/11/14 08:29:01 [verbose] schema state retrieved
2018/11/14 08:29:01 [verbose] preparing nginx prefix directory at /usr/local/kong
2018/11/14 08:29:01 [verbose] SSL enabled, no custom certificate set: using default certificate
2018/11/14 08:29:01 [verbose] generating default SSL certificate and key
2018/11/14 08:29:01 [debug] searching for OpenResty 'nginx' executable
2018/11/14 08:29:01 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.13.6.2'
2018/11/14 08:29:01 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx
2018/11/14 08:29:01 [debug] testing nginx configuration: KONG_NGINX_CONF_CHECK=true /usr/local/openresty/nginx/sbin/nginx -t -p /usr/local/kong -c nginx.conf
2018/11/14 08:29:01 [debug] searching for OpenResty 'nginx' executable
2018/11/14 08:29:01 [debug] /usr/local/openresty/nginx/sbin/nginx -v: 'nginx version: openresty/1.13.6.2'
2018/11/14 08:29:01 [debug] found OpenResty 'nginx' executable at /usr/local/openresty/nginx/sbin/nginx
2018/11/14 08:29:01 [debug] sending signal to pid at: /usr/local/kong/pids/nginx.pid
2018/11/14 08:29:01 [debug] kill -0 `cat /usr/local/kong/pids/nginx.pid` >/dev/null 2>&1
2018/11/14 08:29:01 [debug] starting nginx: /usr/local/openresty/nginx/sbin/nginx -p /usr/local/kong -c nginx.conf
2018/11/14 08:29:02 [debug] nginx started
2018/11/14 08:29:02 [info] Kong started

Kong error logs (<KONG_PREFIX>/logs/error.log)

2018/11/14 08:32:07 [debug] 78#0: *914 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:32:12 [debug] 79#0: *938 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184332.416
2018/11/14 08:32:12 [debug] 79#0: *938 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:32:17 [debug] 77#0: *962 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184337.417
2018/11/14 08:32:17 [debug] 77#0: *962 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:32:22 [debug] 76#0: *987 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184342.42
2018/11/14 08:32:22 [debug] 76#0: *987 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:32:27 [debug] 78#0: *1012 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184347.421
2018/11/14 08:32:27 [debug] 78#0: *1012 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:32:32 [debug] 79#0: *1036 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184352.423
2018/11/14 08:32:32 [debug] 79#0: *1036 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:32:37 [debug] 79#0: *1060 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184357.425
2018/11/14 08:32:37 [debug] 79#0: *1060 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:32:42 [debug] 77#0: *1084 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184362.426
2018/11/14 08:32:42 [debug] 77#0: *1084 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:32:47 [debug] 78#0: *1109 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184367.429
2018/11/14 08:32:47 [debug] 78#0: *1109 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:32:52 [debug] 77#0: *1133 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184372.431
2018/11/14 08:32:52 [debug] 77#0: *1133 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:32:57 [debug] 76#0: *1157 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184377.431
2018/11/14 08:32:57 [debug] 76#0: *1157 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:02 [debug] 79#0: *1182 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184382.435
2018/11/14 08:33:02 [debug] 79#0: *1182 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:33:07 [debug] 77#0: *1206 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184387.436
2018/11/14 08:33:07 [debug] 77#0: *1206 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:12 [debug] 76#0: *1231 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184392.439
2018/11/14 08:33:12 [debug] 76#0: *1231 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:17 [debug] 78#0: *1255 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184397.44
2018/11/14 08:33:17 [debug] 78#0: *1255 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:22 [debug] 79#0: *1280 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184402.442
2018/11/14 08:33:22 [debug] 79#0: *1280 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:27 [debug] 76#0: *1304 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184407.444
2018/11/14 08:33:27 [debug] 76#0: *1304 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:33:32 [debug] 78#0: *1329 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184412.445
2018/11/14 08:33:32 [debug] 78#0: *1329 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:37 [debug] 79#0: *1354 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184417.446
2018/11/14 08:33:37 [debug] 79#0: *1354 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:228: Loading Admin API endpoints
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: response-transformer
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: oauth2
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: acl
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: correlation-id
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: pre-function
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: jwt
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: cors
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: ip-restriction
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: basic-auth
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: key-auth
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: rate-limiting
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: request-transformer
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: http-log
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: file-log
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: hmac-auth
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: ldap-auth
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: datadog
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: tcp-log
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: zipkin
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: post-function
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: request-size-limiting
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: bot-detection
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: syslog
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: loggly
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: azure-functions
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: udp-log
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: response-ratelimiting
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: aws-lambda
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: statsd
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:299: Loading API endpoints for plugin: prometheus
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] init.lua:307: No API endpoints loaded for plugin: request-termination
2018/11/14 08:33:38 [debug] 77#0: *1363 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:33:38 [info] 77#0: *1363 client 10.42.38.87 closed keepalive connection
2018/11/14 08:33:42 [debug] 77#0: *1380 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184422.448
2018/11/14 08:33:42 [debug] 77#0: *1380 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:47 [debug] 77#0: *1404 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184427.451
2018/11/14 08:33:47 [debug] 77#0: *1404 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:52 [debug] 76#0: *1428 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184432.453
2018/11/14 08:33:52 [debug] 76#0: *1428 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:57 [debug] 78#0: *1452 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184437.454
2018/11/14 08:33:57 [debug] 78#0: *1452 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:33:58 [debug] 78#0: *1465 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:33:58 [error] 78#0: *1465 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET / HTTP/1.1", host: "192.168.228.99:8000"
2018/11/14 08:33:58 [error] 78#0: *1466 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:33:59 [debug] 78#0: *1467 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:33:59 [error] 78#0: *1467 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /favicon.ico HTTP/1.1", host: "192.168.228.99:8000", referrer: "http://192.168.228.99:8000/"
2018/11/14 08:33:59 [error] 78#0: *1468 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] base_plugin.lua:24: executing plugin "prometheus": access
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.237.61
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179
2018/11/14 08:34:02 [debug] 77#0: *1481 [lua] base_plugin.lua:24: executing plugin "http-log": access
2018/11/14 08:34:02 [debug] 79#0: *1484 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184442.455
2018/11/14 08:34:02 [debug] 79#0: *1484 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.224.137
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] init.lua:594: balancer(): setting address (try 1): 52.9.82.222:80
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:34:04 [debug] 77#0: *1481 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:34:04 [error] 77#0: *1481 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /ip HTTP/1.1", upstream: "http://52.9.82.222:80/ip", host: "192.168.228.99:8000"
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:34:04 [debug] 79#0: *1513 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:34:04 [error] 79#0: *1513 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /static/main.css HTTP/1.1", host: "192.168.228.99:8000", referrer: "http://192.168.228.99:8000/ip"
2018/11/14 08:34:04 [error] 79#0: *1514 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:34:04 [debug] 78#0: *1515 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:34:04 [error] 78#0: *1515 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /public/style.css HTTP/1.1", host: "192.168.228.99:8000", referrer: "http://192.168.228.99:8000/ip"
2018/11/14 08:34:04 [error] 78#0: *1516 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:34:04 [debug] 79#0: *1517 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:34:04 [error] 79#0: *1517 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /public/style.css HTTP/1.1", host: "192.168.228.99:8000", referrer: "http://192.168.228.99:8000/ip"
2018/11/14 08:34:04 [error] 79#0: *1518 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:20: rewrite(): executing plugin "prometheus": rewrite
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:20: rewrite(): executing plugin "http-log": rewrite
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:28: header_filter(): executing plugin "prometheus": header_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:28: header_filter(): executing plugin "http-log": header_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:32: body_filter(): executing plugin "prometheus": body_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:32: body_filter(): executing plugin "http-log": body_filter
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:36: log(): executing plugin "prometheus": log
2018/11/14 08:34:05 [debug] 78#0: *1519 [lua] base_plugin.lua:36: log(): executing plugin "http-log": log
2018/11/14 08:34:05 [error] 78#0: *1519 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'
        log_by_lua(nginx-kong.conf:125):2: in function <log_by_lua(nginx-kong.conf:125):1> while logging request, client: 10.42.122.134, server: kong, request: "GET /public/favicon.ico HTTP/1.1", host: "192.168.228.99:8000", referrer: "http://192.168.228.99:8000/ip"
2018/11/14 08:34:05 [error] 78#0: *1520 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
        .../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:16: in function <.../local/share/lua/5.1/kong/plugins/prometheus/handler.lua:11>, context: ngx.timer, client: 10.42.122.134, server: 0.0.0.0:8000
2018/11/14 08:34:07 [debug] 76#0: *1529 [lua] cluster_events.lua:221: [cluster_events] polling events from: 1542184141.291 to: 1542184447.457
2018/11/14 08:34:07 [debug] 76#0: *1529 [lua] cluster.lua:428: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.42.89.179

The Kong config {"plugins":{"enabled_in_cluster":["http-log","prometheus"],"available_on_server":{"response-transformer":true,"oauth2":true,"acl":true,"correlation-id":true,"pre-function":true,"jwt":true,"cors":true,"ip-restriction":true,"basic-auth":true,"key-auth":true,"rate-limiting":true,"request-transformer":true,"http-log":true,"file-log":true,"hmac-auth":true,"ldap-auth":true,"datadog":true,"tcp-log":true,"zipkin":true,"post-function":true,"request-size-limiting":true,"bot-detection":true,"syslog":true,"loggly":true,"azure-functions":true,"udp-log":true,"response-ratelimiting":true,"aws-lambda":true,"statsd":true,"prometheus":true,"request-termination":true}},"tagline":"Welcome to kong","configuration":{"plugins":["bundled"],"admin_listen":["0.0.0.0:8001"],"lua_ssl_verify_depth":1,"trusted_ips":{},"prefix":"\/usr\/local\/kong","loaded_plugins":{"response-transformer":true,"request-termination":true,"prometheus":true,"ip-restriction":true,"pre-function":true,"jwt":true,"cors":true,"statsd":true,"basic-auth":true,"key-auth":true,"ldap-auth":true,"aws-lambda":true,"http-log":true,"response-ratelimiting":true,"hmac-auth":true,"request-size-limiting":true,"datadog":true,"tcp-log":true,"zipkin":true,"post-function":true,"bot-detection":true,"acl":true,"loggly":true,"syslog":true,"azure-functions":true,"udp-log":true,"file-log":true,"request-transformer":true,"correlation-id":true,"rate-limiting":true,"oauth2":true},"cassandra_username":"kong","ssl_cert_key":"\/usr\/local\/kong\/ssl\/kong-default.key","dns_resolver":{},"pg_user":"kong","mem_cache_size":"128m","ssl_ciphers":"ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256","nginx_admin_directives":{},"custom_plugins":{},"pg_host":"127.0.0.1","nginx_acc_logs":"\/usr\/local\/kong\/logs\/access.log","proxy_listen":["0.0.0.0:8000","0.0.0.0:8443 ssl"],"client_ssl_cert_default":"\/usr\/local\/kong\/ssl\/kong-default.crt","ssl_cert_csr_default":"\/usr\/local\/kong\/ssl\/kong-default.csr","dns_no_sync":false,"db_update_propagation":1,"nginx_err_logs":"\/usr\/local\/kong\/logs\/error.log","cassandra_port":9042,"dns_order":["LAST","SRV","A","CNAME"],"dns_error_ttl":1,"headers":["server_tokens","latency_tokens"],"cassandra_lb_policy":"RoundRobin","nginx_optimizations":true,"origins":{},"database":"cassandra","pg_database":"kong","nginx_worker_processes":"auto","lua_package_cpath":"","admin_acc_logs":"\/usr\/local\/kong\/logs\/admin_access.log","lua_package_path":".\/?.lua;.\/?\/init.lua;","nginx_pid":"\/usr\/local\/kong\/pids\/nginx.pid","upstream_keepalive":60,"cassandra_contact_points":["cassandra"],"proxy_listeners":[{"ssl":false,"ip":"0.0.0.0","proxy_protocol":false,"port":8000,"http2":false,"listener":"0.0.0.0:8000"},{"ssl":true,"ip":"0.0.0.0","proxy_protocol":false,"port":8443,"http2":false,"listener":"0.0.0.0:8443 ssl"}],"proxy_ssl_enabled":true,"admin_access_log":"logs\/admin_access.log","cassandra_data_centers":["dc1:2","dc2:3"],"enabled_headers":{"latency_tokens":true,"X-Kong-Proxy-Latency":true,"Via":true,"server_tokens":true,"Server":true,"X-Kong-Upstream-Latency":true,"X-Kong-Upstream-Status":false},"cassandra_ssl":false,"admin_listeners":[{"ssl":false,"ip":"0.0.0.0","proxy_protocol":false,"port":8001,"http2":false,"listener":"0.0.0.0:8001"}],"db_resurrect_ttl":30,"cassandra_schema_consensus_timeout":60000,"cassandra_consistency":"ONE","client_max_body_size":"0","admin_error_log":"logs\/error.log","pg_ssl_verify":false,"dns_not_found_ttl":30,"pg_ssl":false,"cassandra_repl_factor":3,"dns_stale_ttl":4,"cassandra_repl_strategy":"SimpleStrategy","nginx_conf":"\/usr\/local\/kong\/nginx.conf","db_update_frequency":5,"nginx_http_directives":[{"value":"prometheus_metrics 5m","name":"lua_shared_dict"}],"pg_timeout":60000,"kong_env":"\/usr\/local\/kong\/.kong_env","real_ip_header":"X-Real-IP","dns_hostsfile":"\/etc\/hosts","log_level":"debug","error_default_type":"text\/plain","ssl_cert":"\/usr\/local\/kong\/ssl\/kong-default.crt","proxy_access_log":"logs\/access.log","admin_ssl_cert_key_default":"\/usr\/local\/kong\/ssl\/admin-kong-default.key","cassandra_ssl_verify":false,"client_ssl":false,"ssl_cipher_suite":"modern","real_ip_recursive":"off","proxy_error_log":"logs\/error.log","client_ssl_cert_key_default":"\/usr\/local\/kong\/ssl\/kong-default.key","nginx_daemon":"on","anonymous_reports":true,"cassandra_timeout":60000,"nginx_proxy_directives":{},"pg_port":5432,"nginx_kong_conf":"\/usr\/local\/kong\/nginx-kong.conf","client_body_buffer_size":"8k","lua_socket_pool_size":30,"admin_ssl_cert_default":"\/usr\/local\/kong\/ssl\/admin-kong-default.crt","db_cache_ttl":0,"cassandra_keyspace":"kong","ssl_cert_default":"\/usr\/local\/kong\/ssl\/kong-default.crt","ssl_cert_key_default":"\/usr\/local\/kong\/ssl\/kong-default.key","admin_ssl_enabled":false},"version":"1.0.0rc2","node_id":"baa91c6a-1c2c-4f3d-8c63-f997476c8970","lua_version":"LuaJIT 2.1.0-beta3","prng_seeds":{"pid: 79":361652051992,"pid: 77":146101302524,"pid: 68":176112203523,"pid: 78":332063021725,"pid: 76":223531004135},"timers":{"pending":4,"running":0},"hostname":"d4997caa3065"}

rahimimo commented 6 years ago

It is visible from Kong error.logs

hishamhm commented 6 years ago

@hbagdi this is the service is nil bug that has already been fixed after 1.0rc2 was released, right?

2018/11/14 08:33:58 [error] 78#0: *1466 lua entry thread aborted: runtime error: ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua:50: attempt to index field 'service' (a nil value)
stack traceback:
coroutine 0:
        ...local/share/lua/5.1/kong/plugins/prometheus/exporter.lua: in function 'log'
hishamhm commented 6 years ago

@rahimimo Thank you for the logs! Indeed it looks like a bug on the plugin's end.

2018/11/14 08:34:04 [error] 79#0: *1517 failed to run log_by_lua*: /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: table index is nil
stack traceback:
        /usr/local/share/lua/5.1/kong/plugins/http-log/handler.lua:77: in function 'log'
        /usr/local/share/lua/5.1/kong/init.lua:727: in function 'log'

@rahimimo, could you try changing line 50 in kong/plugins/http-log/handler.lua from

  local route_id = conf.route_id or conf.api_id

to

  local route_id = conf.route_id or conf.api_id or "global"

Please let us know if that fixes the issue with http-log.

jeremyjpj0916 commented 6 years ago

I experienced this bug when using some of the original logic too, fixed here: https://github.com/Optum/kong-splunk-log/blob/master/src/handler.lua , I got around it same way hishamhm suggested, except I called mine "notfound" because even if the proxy path does not exist I still want http-log to process the tx for logging purposes. Note this happens to me when plugin is enabled globally.

jeremyjpj0916 commented 6 years ago

@rahimimo @hishamhm , I think this can be closed now with my PR merged this will be fixed on next major release.

hishamhm commented 6 years ago

@jeremyjpj0916 Yes, indeed! Thank you!