search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/#kong-community
Apache License 2.0
38.95k stars 4.78k forks source link

[kong 2.1.3][DB-less] runtime error when reloading #6359

Closed debu99 closed 3 years ago

debu99 commented 4 years ago

We got this error after upgrade to 2.1.3 from 2.0.4

2020/09/17 01:52:52 [crit] 32221#0: *1399655 [lua] init.lua:591: init_worker(): worker initialization error: failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating; this node must be restarted, context: init_worker_by_lua*
2020/09/17 01:52:54 [notice] 19185#0: exit
2020/09/17 01:52:58 [notice] 29558#0: exiting
2020/09/17 01:52:58 [notice] 29558#0: exit
2020/09/17 01:53:02 [notice] 28310#0: using the "epoll" event method
2020/09/17 01:53:02 [notice] 28310#0: start worker processes
2020/09/17 01:53:02 [notice] 28310#0: start worker process 32231
2020/09/17 01:53:03 [notice] 28310#0: signal 17 (SIGCHLD) received from 32222
2020/09/17 01:53:03 [notice] 28310#0: worker process 19185 exited with code 0
2020/09/17 01:53:03 [notice] 28310#0: worker process 29558 exited with code 0
2020/09/17 01:53:02 [crit] 32231#0: *1399698 [lua] init.lua:591: init_worker(): worker initialization error: failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating; this node must be restarted, context: init_worker_by_lua*
2020/09/17 01:53:04 [notice] 28310#0: signal 29 (SIGIO) received
2020/09/17 01:53:04 [notice] 28310#0: signal 29 (SIGIO) received
2020/09/17 01:53:04 [alert] 32231#0: *1399701 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 9.8.7.6, server: 0.0.0.0:6443

> 
2020/09/17 03:36:21 [error] 32231#0: *1409893 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/handler.lua:1094: attempt to index local 'router' (a nil value)
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/runloop/handler.lua: in function 'before'
    /usr/local/share/lua/5.1/kong/init.lua:742: in function 'access'
    access_by_lua(nginx-kong.conf:95):2: in main chunk, client: 6.7.8.9, server: kong, request: "POST /api/v3/user-service/v3/sg/en/mobile/devices/39FF20D7-535E-4D00-A020-9EE26AC9D768com.domain.selfcare.qa/update HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:21 [alert] 32231#0: *1409893 [lua] init.lua:1172: handle_error(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), client: 6.7.8.9, server: kong, request: "POST /api/v3/user-service/v3/sg/en/mobile/devices/39FF20D7-535E-4D00-A020-9EE26AC9D768com.domain.selfcare.qa/update HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:45 [alert] 32231#0: *1409920 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 5.6.7.8, server: 0.0.0.0:6443
2020/09/17 03:36:45 [error] 32231#0: *1409919 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/handler.lua:1094: attempt to index local 'router' (a nil value)
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/runloop/handler.lua: in function 'before'
    /usr/local/share/lua/5.1/kong/init.lua:742: in function 'access'
    access_by_lua(nginx-kong.conf:95):2: in main chunk, client: 5.6.7.8, server: kong, request: "GET /api/v3/user-service/v4/sg/en/mobile/users/token HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:45 [alert] 32231#0: *1409919 [lua] init.lua:1172: handle_error(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), client: 5.6.7.8, server: kong, request: "GET /api/v3/user-service/v4/sg/en/mobile/users/token HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:45 [alert] 32231#0: *1409922 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 5.6.7.8, server: 0.0.0.0:6443
2020/09/17 03:36:45 [error] 32231#0: *1409921 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/runloop/handler.lua:1094: attempt to index local 'router' (a nil value)
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/runloop/handler.lua: in function 'before'
    /usr/local/share/lua/5.1/kong/init.lua:742: in function 'access'
    access_by_lua(nginx-kong.conf:95):2: in main chunk, client: 5.6.7.8, server: kong, request: "POST /api/v3/user-service/v3/sg/en/mobile/devices/C4ED25BE-5C1B-4198-AF83-2C1FE3593E02com.domain.selfcare.qa/update HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:45 [alert] 32231#0: *1409921 [lua] init.lua:1172: handle_error(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), client: 5.6.7.8, server: kong, request: "POST /api/v3/user-service/v3/sg/en/mobile/devices/C4ED25BE-5C1B-4198-AF83-2C1FE3593E02com.domain.selfcare.qa/update HTTP/1.1", host: "test.mydomain.com:6443"
2020/09/17 03:36:49 [alert] 32231#0: *1409928 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 1.2.3.4, server: 0.0.0.0:6443
2020/09/17 03:36:49 [alert] 32231#0: *1409930 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 1.2.3.4, server: 0.0.0.0:6443
2020/09/17 03:36:55 [alert] 32231#0: *1409938 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 2.3.4.5, server: 0.0.0.0:6443
2020/09/17 03:36:55 [alert] 32231#0: *1409940 [lua] init.lua:666: ssl_certificate(): unsafe request processing due to earlier initialization errors; this node must be restarted (failed to load declarative config file: error building initial router: could not load routes: [off] stale data detected while paginating), context: ssl_certificate_by_lua*, client: 2.3.4.5, server: 0.0.0.0:6443
debu99 commented 4 years ago

I think this is caused by kong reload

bungle commented 4 years ago

I think this is caused by kong reload

Yes, most definitely by that. The reload is problematic in many ways. E.g. you cannot pass environment vars and it is hard to see what was passed previously.

kikito commented 4 years ago

Hello @debu99 could you try with kong 2.1.4

debu99 commented 4 years ago

yes, already updated for 1 day, let's monitor for 1 week

kikito commented 3 years ago

@debu99 How is it looking? Did the update solve the problem?

debu99 commented 3 years ago

thanks,looks okay, we can close the issue