Kong :2.0.5 DP node cannot connect to CP node in hybrid mode, but works normally in Kong2.0.4

[xinfeingxia85]

Summary

Everything worked fine with the Kong2.04 version, but after I upgraded to Kong2.0.5, DP Node was unable to get configuration information from CP Node. If CP Node's version is KONG :2.0.4 and DP Node's version is 2.0.5, everything works fine.

Additional Details & Logs

• kong error

  2020/07/15 13:40:37 [error] 22#0: *4 [lua] clustering.lua:149: error while receiving frame from control plane: failed to receive the first 2 bytes: closed, context: ngx.timer
  2020/07/15 13:40:47 [error] 22#0: *7320 [lua] clustering.lua:165: unable to update running config: bad config received from control plane, context: ngx.timer
  2020/07/15 13:41:06 [error] 22#0: *4 [lua] clustering.lua:96: send_ping(): unable to ping control plane node: fatal error already happened, context: ngx.timer

• Run environment

  Client: Docker Engine - Community
  Version:           19.03.5
  API version:       1.40
  Go version:        go1.12.12
  Git commit:        633a0ea
  Built:             Wed Nov 13 07:22:34 2019
  OS/Arch:           darwin/amd64
  Experimental:      false

• Server: Docker Engine - Community

  Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     false
  containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
  runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
  docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[hutchic]

Could you provide information on how you ran into this ie docker-compose or k8s yaml manifest?

We recently re-pushed our 2.0.5 images which included a small entrypoint change that may be related if you could re-run with 2.0.5 that'd be appreciated 👍

[dndx]

Hello @xinfeingxia85,

Thanks for your report. However I ran the 2.0.5 version locally and could not reproduce the issue you are describing. Would you mind sharing the full error log from both the CP and DP nodes to aid debugging?

[gszr]

Will be helpful to know the exact environment variables / configurations that were passed to both CP and DP containers as well.

[huaerfan]

When I upgraded custom plugins on DP node and restart DP node, this happen. Both CP and DP version are 2.0.4

2020/07/17 03:08:58 [notice] 28341#0: openresty/1.15.8.3 2020/07/17 03:08:58 [notice] 28341#0: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 2020/07/17 03:08:58 [notice] 28341#0: OS: Linux 3.10.0-862.3.2.el7.x86_64

2020/07/17 03:09:28 [error] 28350#0: 4 [lua] clustering.lua:149: error while receiving frame from control plane: failed to receive the first 2 bytes: closed, context: ngx.timer 2020/07/17 03:09:58 [error] 28350#0: 4 [lua] clustering.lua:96: send_ping(): unable to ping control plane node: fatal error already happened, context: ngx.timer 2020/07/17 03:10:07 [error] 28350#0: 385 [lua] clustering.lua:149: error while receiving frame from control plane: failed to receive the first 2 bytes: closed, context: ngx.timer 2020/07/17 03:10:37 [error] 28350#0: 385 [lua] clustering.lua:96: send_ping(): unable to ping control plane node: fatal error already happened, context: ngx.timer

[dndx]

@huaerfan Thanks for the log. However, it seems that they are only DP logs. Could you provide the CP error logs as well? Without them it is difficult to understand what happened that caused the connection to broke.

[huaerfan]

Thanks @dndx ,The problem has already been solved, the reason was Configured Entities size more than 65535 byte which as default setting in version 2.0.4 and don't compress data before transmission, In version 2.0.5, the default setting is 4M, and Compressed with zip format before transmission.

Thanks again.

[dndx]

@huaerfan That's great news to hear, thanks for the feedback!

[andreaguiarvia]

@huaerfan Hello man, i have the same problem, do you can describe more the solution?