Kong PDK kong.service.set_target() not properly setting the "Host" header

[jonhy-bigodes]

Summary

While developing a plugin that changes the upstream target host/port it was noticed that the "Host" header that is set while invoking PDK function kong.service.set_target() is not setting the port subcomponent, only the host part. This is not conforming with RFC7230, Section 5.4. When a target service uses an intermediate load balancer (eg: HAProxy) and the service uses a non-standard http/https port, the service request call fails at the load balancer level since the "Host" header is missing the port information (set by kong.service.set_target()).

Steps To Reproduce

1. Create a route and service
2. Develop a new plugin or use serverless functions plugin "pre-function" to change both the upstream host and port (eg:10000) using PDK kong.service.set_target()
3. Activate the plugin over step 1. route or service
4. Setup a mock server (one-liner in Linux: while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; } | nc -4 -l 10000 -q 1; done)
5. Hit Kong route configured in step 1 with a request. Check mock server headers input, it will miss the port subcomponent. This causes issues when load balancers are used as the service upstream address.

Additional Details & Logs

• Kong version 1.2.3 (bug should happen in any ensuing version)
• Kong debug-level startup logs

  2020/05/25 08:21:33 [debug] 1#0: [lua] globalpatches.lua:13: installing the globalpatches
  2020/05/25 08:21:33 [debug] 1#0: [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 1#0: [lua] globalpatches.lua:237: randomseed(): random seed: 115129233391 for worker nb 0
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:441: init(): [dns-client] (re)configuring dns client
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:446: init(): [dns-client] staleTtl = 4
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:449: init(): [dns-client] validTtl = nil
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:453: init(): [dns-client] noSynchronisation = false
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:472: init(): [dns-client] query order = LAST, SRV, A, CNAME
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:512: init(): [dns-client] adding A-record from 'hosts' file: 1481dfceb1e2 = 172.21.0.3
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:512: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:571: init(): [dns-client] nameserver 127.0.0.11
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:576: init(): [dns-client] attempts = 5
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] timeout = 2000 ms
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:589: init(): [dns-client] ndots = 0
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:591: init(): [dns-client] search = lan
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] badTtl = 30 s
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:599: init(): [dns-client] emptyTtl = 1 s
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:127: check_db_against_config(): Discovering used plugins
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:441: init(): [dns-client] (re)configuring dns client
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:446: init(): [dns-client] staleTtl = 4
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:449: init(): [dns-client] validTtl = nil
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:453: init(): [dns-client] noSynchronisation = false
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:472: init(): [dns-client] query order = LAST, SRV, A, CNAME
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:512: init(): [dns-client] adding A-record from 'hosts' file: 1481dfceb1e2 = 172.21.0.3
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:512: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:527: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:571: init(): [dns-client] nameserver 127.0.0.11
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:576: init(): [dns-client] attempts = 5
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] timeout = 2000 ms
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:589: init(): [dns-client] ndots = 0
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:591: init(): [dns-client] search = lan
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] badTtl = 30 s
  2020/05/25 08:21:33 [debug] 1#0: [lua] client.lua:599: init(): [dns-client] emptyTtl = 1 s
  2020/05/25 08:21:33 [info] 1#0: [lua] mesh.lua:64: init(): initialising cluster ca...
  2020/05/25 08:21:33 [info] 1#0: [lua] mesh.lua:137: init(): cluster ca initialised
  2020/05/25 08:21:33 [debug] 1#0: [lua] mesh.lua:138: init(): mesh alpn is 'com.konghq/service-mesh/1/NTInyjbeH07XfV8xqKHEF/zS/q8SqZD7uFkte2WHSNw='
  2020/05/25 08:21:33 [debug] 1#0: [lua] mesh.lua:139: init(): node certificate generated, fingerprint is 'CfTefMNxAgOriBKGjd+CPx3sqt6KOBE6JjiHWepgAZ4='
  2020/05/25 08:21:33 [debug] 1#0: [lua] mesh.lua:188: init(): SSL_CTX* mesh ALPN selection setup complete
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: correlation-id
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: pre-function
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: cors
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: fflink-dynamic-upstream
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: ldap-auth
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: loggly
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: hmac-auth
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'hmac-auth.hmacauth_credentials'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: zipkin
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: request-size-limiting
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: azure-functions
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: request-transformer
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: oauth2
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'oauth2.oauth2_credentials'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'oauth2.oauth2_authorization_codes'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'oauth2.oauth2_tokens'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: response-transformer
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: ip-restriction
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: statsd
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: jwt
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'jwt.jwt_secrets'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: proxy-cache
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: jwt-validation
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: basic-auth
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'basic-auth.basicauth_credentials'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: key-auth
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'key-auth.keyauth_credentials'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: http-log
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: fflink-forward-proxy
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: datadog
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: tcp-log
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: fflink-keymap-oidc-auth
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'fflink-keymap-oidc-auth.fflink_keymap_oidc_credentials'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: post-function
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: prometheus
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: acl
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:205: loader_fn(): Loading custom plugin entity: 'acl.acls'
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: kubernetes-sidecar-injector
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: syslog
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: file-log
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: udp-log
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: response-ratelimiting
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: aws-lambda
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: bot-detection
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: rate-limiting
  2020/05/25 08:21:33 [debug] 1#0: [lua] plugins.lua:243: load_plugin(): Loading plugin: request-termination
  2020/05/25 08:21:33 [notice] 1#0: using the "epoll" event method
  2020/05/25 08:21:33 [notice] 1#0: openresty/1.13.6.2
  2020/05/25 08:21:33 [notice] 1#0: built by gcc 9.2.0 (Alpine 9.2.0) 
  2020/05/25 08:21:33 [notice] 1#0: OS: Linux 4.18.0-13-generic
  2020/05/25 08:21:33 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
  2020/05/25 08:21:33 [notice] 1#0: start worker processes
  2020/05/25 08:21:33 [notice] 1#0: start worker process 32
  2020/05/25 08:21:33 [notice] 1#0: start worker process 33
  2020/05/25 08:21:33 [notice] 1#0: start worker process 34
  2020/05/25 08:21:33 [notice] 1#0: start worker process 35
  2020/05/25 08:21:33 [notice] 1#0: start worker process 36
  2020/05/25 08:21:33 [notice] 1#0: start worker process 37
  2020/05/25 08:21:33 [notice] 1#0: start worker process 38
  2020/05/25 08:21:33 [notice] 1#0: start worker process 39
  2020/05/25 08:21:33 [debug] 32#0: *1 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 32#0: *1 [lua] globalpatches.lua:237: randomseed(): random seed: 209981003511 for worker nb 0
  2020/05/25 08:21:33 [debug] 33#0: *2 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 33#0: *2 [lua] globalpatches.lua:237: randomseed(): random seed: 221193765812 for worker nb 1
  2020/05/25 08:21:33 [debug] 35#0: *3 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 35#0: *3 [lua] globalpatches.lua:237: randomseed(): random seed: 932132461281 for worker nb 3
  2020/05/25 08:21:33 [debug] 34#0: *4 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 34#0: *4 [lua] globalpatches.lua:237: randomseed(): random seed: 243211452462 for worker nb 2
  2020/05/25 08:21:33 [debug] 37#0: *5 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 37#0: *5 [lua] globalpatches.lua:237: randomseed(): random seed: 652061258592 for worker nb 5
  2020/05/25 08:21:33 [debug] 33#0: *2 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=33, data=nil
  2020/05/25 08:21:33 [debug] 32#0: *1 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=32, data=nil
  2020/05/25 08:21:33 [debug] 33#0: *2 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=32, data=nil
  2020/05/25 08:21:33 [debug] 36#0: *6 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 36#0: *6 [lua] globalpatches.lua:237: randomseed(): random seed: 201119143218 for worker nb 4
  2020/05/25 08:21:33 [debug] 35#0: *3 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=35, data=nil
  2020/05/25 08:21:33 [debug] 34#0: *4 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=34, data=nil
  2020/05/25 08:21:33 [debug] 37#0: *5 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=37, data=nil
  2020/05/25 08:21:33 [debug] 36#0: *6 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=36, data=nil
  2020/05/25 08:21:33 [debug] 38#0: *7 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 38#0: *7 [lua] globalpatches.lua:237: randomseed(): random seed: 141732046518 for worker nb 6
  2020/05/25 08:21:33 [debug] 38#0: *7 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=38, data=nil
  2020/05/25 08:21:33 [notice] 32#0: *1 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'services' into the cache ..., context: init_worker_by_lua*
  2020/05/25 08:21:33 [debug] 35#0: *3 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [debug] 37#0: *5 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [debug] 33#0: *2 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [debug] 36#0: *6 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [debug] 34#0: *4 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [notice] 32#0: *1 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'services' into the cache (in 39ms), context: init_worker_by_lua*
  2020/05/25 08:21:33 [notice] 32#0: *1 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'plugins' into the cache ..., context: init_worker_by_lua*
  2020/05/25 08:21:33 [debug] 39#0: *14 [lua] globalpatches.lua:211: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
  2020/05/25 08:21:33 [debug] 39#0: *14 [lua] globalpatches.lua:237: randomseed(): random seed: 100817911485 for worker nb 7
  2020/05/25 08:21:33 [debug] 39#0: *14 [lua] events.lua:192: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=39, data=nil
  2020/05/25 08:21:33 [debug] 38#0: *7 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [notice] 32#0: *1 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'plugins' into the cache (in 15ms), context: init_worker_by_lua*
  2020/05/25 08:21:33 [debug] 32#0: *1 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [debug] 39#0: *14 [lua] base_plugin.lua:13: init_worker(): executing plugin "fflink-keymap-oidc-auth": init_worker
  2020/05/25 08:21:33 [notice] 32#0: *17 [lua] cache_warmup.lua:24: warming up DNS entries ..., context: ngx.timer
  2020/05/25 08:21:33 [debug] 37#0: *8 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 38#0: *16 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 32#0: *18 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 39#0: *21 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 33#0: *11 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 36#0: *12 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 34#0: *13 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:33 [debug] 35#0: *15 [lua] balancer.lua:751: init(): initialized 0 balancer(s), 0 error(s)
  2020/05/25 08:21:34 [notice] 32#0: *17 [lua] cache_warmup.lua:34: finished warming up DNS entries' into the cache (in 854ms), context: ngx.timer

• Kong error logs: no errors are shown at Kong level
• Kong configuration:

  {
  "plugins": {
      "enabled_in_cluster": [
          "fflink-forward-proxy",
          "fflink-dynamic-upstream",
          "request-transformer"
      ],
      "available_on_server": {
          "correlation-id": true,
          "pre-function": true,
          "cors": true,
          "fflink-dynamic-upstream": true,
          "ldap-auth": true,
          "loggly": true,
          "hmac-auth": true,
          "zipkin": true,
          "request-size-limiting": true,
          "azure-functions": true,
          "request-transformer": true,
          "oauth2": true,
          "response-transformer": true,
          "ip-restriction": true,
          "statsd": true,
          "jwt": true,
          "proxy-cache": true,
          "jwt-validation": true,
          "basic-auth": true,
          "key-auth": true,
          "http-log": true,
          "fflink-forward-proxy": true,
          "datadog": true,
          "tcp-log": true,
          "fflink-keymap-oidc-auth": true,
          "post-function": true,
          "prometheus": true,
          "acl": true,
          "kubernetes-sidecar-injector": true,
          "syslog": true,
          "file-log": true,
          "udp-log": true,
          "response-ratelimiting": true,
          "aws-lambda": true,
          "bot-detection": true,
          "rate-limiting": true,
          "request-termination": true
      }
  },
  "tagline": "Welcome to kong",
  "configuration": {
      "error_default_type": "text/plain",
      "admin_listen": [
          "0.0.0.0:8001",
          "0.0.0.0:8444 ssl"
      ],
      "proxy_access_log": "/dev/stdout",
      "trusted_ips": {},
      "prefix": "/usr/local/kong",
      "loaded_plugins": {
          "correlation-id": true,
          "pre-function": true,
          "cors": true,
          "fflink-dynamic-upstream": true,
          "ldap-auth": true,
          "loggly": true,
          "hmac-auth": true,
          "zipkin": true,
          "request-size-limiting": true,
          "azure-functions": true,
          "request-transformer": true,
          "oauth2": true,
          "response-transformer": true,
          "syslog": true,
          "statsd": true,
          "jwt": true,
          "proxy-cache": true,
          "jwt-validation": true,
          "basic-auth": true,
          "key-auth": true,
          "http-log": true,
          "prometheus": true,
          "datadog": true,
          "tcp-log": true,
          "rate-limiting": true,
          "post-function": true,
          "bot-detection": true,
          "acl": true,
          "kubernetes-sidecar-injector": true,
          "ip-restriction": true,
          "response-ratelimiting": true,
          "udp-log": true,
          "file-log": true,
          "aws-lambda": true,
          "fflink-keymap-oidc-auth": true,
          "fflink-forward-proxy": true,
          "request-termination": true
      },
      "cassandra_username": "kong",
      "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
      "admin_ssl_cert_key": "/usr/local/kong/ssl/admin-kong-default.key",
      "dns_resolver": {},
      "pg_user": "kong",
      "pg_password": "******",
      "cassandra_data_centers": [
          "dc1:2",
          "dc2:3"
      ],
      "nginx_admin_directives": {},
      "nginx_http_directives": [
          {
              "value": "prometheus_metrics 5m",
              "name": "lua_shared_dict"
          }
      ],
      "pg_host": "db",
      "nginx_acc_logs": "/usr/local/kong/logs/access.log",
      "pg_semaphore_timeout": 60000,
      "proxy_listen": [
          "0.0.0.0:8000",
          "0.0.0.0:8443 ssl"
      ],
      "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
      "cassandra_ssl": false,
      "db_update_frequency": 5,
      "db_update_propagation": 0,
      "stream_listen": [
          "off"
      ],
      "nginx_err_logs": "/usr/local/kong/logs/error.log",
      "cassandra_port": 9042,
      "dns_order": [
          "LAST",
          "SRV",
          "A",
          "CNAME"
      ],
      "dns_error_ttl": 1,
      "headers": [
          "server_tokens",
          "latency_tokens"
      ],
      "cassandra_lb_policy": "RequestRoundRobin",
      "nginx_optimizations": true,
      "pg_timeout": 5000,
      "database": "postgres",
      "pg_database": "kong",
      "nginx_worker_processes": "auto",
      "lua_package_cpath": "",
      "admin_ssl_cert": "/usr/local/kong/ssl/admin-kong-default.crt",
      "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
      "real_ip_header": "X-Real-IP",
      "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
      "lua_package_path": "./?.lua;./?/init.lua;",
      "nginx_pid": "/usr/local/kong/pids/nginx.pid",
      "upstream_keepalive": 60,
      "nginx_conf": "/usr/local/kong/nginx.conf",
      "router_consistency": "strict",
      "dns_no_sync": false,
      "origins": {},
      "admin_access_log": "/dev/stdout",
      "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
      "client_ssl": false,
      "proxy_listeners": [
          {
              "transparent": false,
              "ssl": false,
              "ip": "0.0.0.0",
              "proxy_protocol": false,
              "port": 8000,
              "http2": false,
              "listener": "0.0.0.0:8000"
          },
          {
              "transparent": false,
              "ssl": true,
              "ip": "0.0.0.0",
              "proxy_protocol": false,
              "port": 8443,
              "http2": false,
              "listener": "0.0.0.0:8443 ssl"
          }
      ],
      "proxy_ssl_enabled": true,
      "stream_listeners": {},
      "db_cache_warmup_entities": [
          "services",
          "plugins"
      ],
      "enabled_headers": {
          "latency_tokens": true,
          "X-Kong-Proxy-Latency": true,
          "Via": true,
          "server_tokens": true,
          "Server": true,
          "X-Kong-Upstream-Latency": true,
          "X-Kong-Upstream-Status": false
      },
      "plugins": [
          "bundled",
          "fflink-keymap-oidc-auth",
          "fflink-dynamic-upstream",
          "fflink-forward-proxy",
          "jwt-validation"
      ],
      "ssl_ciphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256",
      "db_resurrect_ttl": 30,
      "nginx_proxy_directives": {},
      "cassandra_consistency": "ONE",
      "client_max_body_size": "0",
      "admin_error_log": "/dev/stderr",
      "pg_ssl_verify": false,
      "dns_not_found_ttl": 30,
      "pg_ssl": false,
      "lua_ssl_verify_depth": 1,
      "ssl_cipher_suite": "modern",
      "cassandra_repl_strategy": "SimpleStrategy",
      "proxy_error_log": "/dev/stderr",
      "kong_env": "/usr/local/kong/.kong_env",
      "db_cache_ttl": 0,
      "pg_max_concurrent_queries": 0,
      "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
      "cassandra_schema_consensus_timeout": 10000,
      "dns_hostsfile": "/etc/hosts",
      "admin_listeners": [
          {
              "transparent": false,
              "ssl": false,
              "ip": "0.0.0.0",
              "proxy_protocol": false,
              "port": 8001,
              "http2": false,
              "listener": "0.0.0.0:8001"
          },
          {
              "transparent": false,
              "ssl": true,
              "ip": "0.0.0.0",
              "proxy_protocol": false,
              "port": 8444,
              "http2": false,
              "listener": "0.0.0.0:8444 ssl"
          }
      ],
      "dns_stale_ttl": 4,
      "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
      "cassandra_timeout": 5000,
      "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
      "cassandra_ssl_verify": false,
      "mem_cache_size": "128m",
      "log_level": "debug",
      "real_ip_recursive": "off",
      "cassandra_repl_factor": 1,
      "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
      "nginx_daemon": "off",
      "anonymous_reports": true,
      "nginx_sproxy_directives": {},
      "nginx_stream_directives": {},
      "pg_port": 5432,
      "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
      "client_body_buffer_size": "8k",
      "ssl_preread_enabled": true,
      "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
      "cassandra_contact_points": [
          "db"
      ],
      "cassandra_keyspace": "kong",
      "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
      "lua_socket_pool_size": 30,
      "admin_ssl_enabled": true
  },
  "version": "1.2.3",
  "node_id": "3efaf968-52d7-4a53-a634-fad1bc3e05e4",
  "lua_version": "LuaJIT 2.1.0-beta3",
  "prng_seeds": {
      "pid: 32": 140451738671,
      "pid: 35": 412040157166,
      "pid: 1": 352019116940,
      "pid: 34": 133351031352,
      "pid: 38": 222201605922,
      "pid: 37": 115133145240,
      "pid: 33": 225331783044,
      "pid: 31": 931562712013,
      "pid: 36": 872282201911
  },
  "timers": {
      "pending": 7,
      "running": 0
  },
  "hostname": "1481dfceb1e2"
  }

• Operating system: Ubuntu 18.04, tests done using kong:1.2.3 docker image

[kikito]

Hello, thanks for reporting this. Could you please try with a more recent version of Kong?

[jonhy-bigodes]

Sure, no problem. With the latest version the bug still happens. The problem is located at the PDK code https://github.com/Kong/kong/blob/master/kong/pdk/service.lua (line 99): ngx.var.upstream_host = host

This _upstreamhost variable drives the "Host" header and is missing the port subcomponent. Just changing code to ngx.var.upstream_host = host .. ":" .. port should fix it.

In the meantime, the workaround is setting the "Host" header programmatically after the _kong.service.settarget() call.