Open NICK-DUAN opened 7 months ago
You can add the suggested argument to the proxy container. We don't expose args in values.yaml (most everything is controlled through environment variables instead, but the verbose flags aren't), so you'd need to run kubectl edit deploy DEPLOYMENT_NAME
, find the proxy container (search for containers:
), and add
args:
- --vv
Given such a large version jump, you may want to consider simply starting from a fresh install and adding settings you need on top of that. If your routing configuration is all handled through Ingresses/Services/etc. you don't strictly need to upgrade the actual Deployment, since the controller will restore it.
There are changes to Kubernetes and chart configuration that you'll need to perform regardless, however.
https://github.com/Kong/charts/blob/main/charts/kong/UPGRADE.md covers chart values.yaml changes.
https://docs.konghq.com/kubernetes-ingress-controller/2.11.x/guides/upgrade/ covers steps to upgrade from KIC 1.x to 2.x. https://docs.konghq.com/kubernetes-ingress-controller/3.0.x/guides/migrate/kongingress/ and the other guides in that section cover going from KIC 2.x to 3.x
https://docs.konghq.com/kubernetes-ingress-controller/2.11.x/guides/upgrade-kong-3x/ covers going from Kong 2.x to 3.x
You can add the suggested argument to the proxy container. We don't expose args in values.yaml (most everything is controlled through environment variables instead, but the verbose flags aren't), so you'd need to run
kubectl edit deploy DEPLOYMENT_NAME
, find the proxy container (search forcontainers:
), and addargs: - --vv
Given such a large version jump, you may want to consider simply starting from a fresh install and adding settings you need on top of that. If your routing configuration is all handled through Ingresses/Services/etc. you don't strictly need to upgrade the actual Deployment, since the controller will restore it.
There are changes to Kubernetes and chart configuration that you'll need to perform regardless, however.
https://github.com/Kong/charts/blob/main/charts/kong/UPGRADE.md covers chart values.yaml changes.
https://docs.konghq.com/kubernetes-ingress-controller/2.11.x/guides/upgrade/ covers steps to upgrade from KIC 1.x to 2.x. https://docs.konghq.com/kubernetes-ingress-controller/3.0.x/guides/migrate/kongingress/ and the other guides in that section cover going from KIC 2.x to 3.x
https://docs.konghq.com/kubernetes-ingress-controller/2.11.x/guides/upgrade-kong-3x/ covers going from Kong 2.x to 3.x
after add args, run helm -n kong install kong kong/kong -f ./charts/config.yaml --debug
i got this in proxy
container(args=["kong", "docker-start", "--vv"]
):
Error:
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28:
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: in function 'load_conf'
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:85: in function </usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:84>
[C]: in function 'xpcall'
/usr/local/bin/kong:99: in function 'file_gen'
init_worker_by_lua(nginx.conf:189):51: in function <init_worker_by_lua(nginx.conf:189):49>
[C]: in function 'xpcall'
init_worker_by_lua(nginx.conf:189):58: in function <init_worker_by_lua(nginx.conf:189):56>
And this time i use this custom-values:
env:
admin_listen: "127.0.0.1:8001"
log_level: debug
admin:
enabled: true
http:
enabled: true
tls:
enabled: false
And I create a new namespace to deploy kong, i think there are no distractions this time
after add args, run
helm -n kong install kong kong/kong -f ./charts/config.yaml --debug
i got this inproxy
container(args=["kong", "docker-start", "--vv"]
):Error: /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: stack traceback: [C]: in function 'assert' /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: in function 'load_conf' /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:85: in function </usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:84> [C]: in function 'xpcall' /usr/local/bin/kong:99: in function 'file_gen' init_worker_by_lua(nginx.conf:189):51: in function <init_worker_by_lua(nginx.conf:189):49> [C]: in function 'xpcall' init_worker_by_lua(nginx.conf:189):58: in function <init_worker_by_lua(nginx.conf:189):56>
....
And I create a new namespace to deploy kong, i think there are no distractions this time
And it doesn't have any other error detail above? It looks like there should be more detailed information about why https://github.com/Kong/kong/blob/3.5.0/kong/cmd/utils/inject_confs.lua#L28 failed.
That code is broadly concerned with initializing some files Kong creates on start, but I can't think of obvious reasons that would fail or be tied to an upgrade, as it's created fresh for each container replica. The chart changes related to that were to create a dedicated EmptyDir volume when we started using read-only root filesystems, but your Deployment isn't missing that and isn't trying to use some other prefix location, so I wouldn't expect any issues there.
You could file an upstream issue in https://github.com/Kong/kong to try and see if they can determine why the prefix initializer is reporting an error but no details. but if you are indeed able to clear the issue by starting from a fresh install that's probably quicker.
after add args, run
helm -n kong install kong kong/kong -f ./charts/config.yaml --debug
i got this inproxy
container(args=["kong", "docker-start", "--vv"]
):Error: /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: stack traceback: [C]: in function 'assert' /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: in function 'load_conf' /usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:85: in function </usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:84> [C]: in function 'xpcall' /usr/local/bin/kong:99: in function 'file_gen' init_worker_by_lua(nginx.conf:189):51: in function <init_worker_by_lua(nginx.conf:189):49> [C]: in function 'xpcall' init_worker_by_lua(nginx.conf:189):58: in function <init_worker_by_lua(nginx.conf:189):56>
.... And I create a new namespace to deploy kong, i think there are no distractions this time
And it doesn't have any other error detail above? It looks like there should be more detailed information about why https://github.com/Kong/kong/blob/3.5.0/kong/cmd/utils/inject_confs.lua#L28 failed.
That code is broadly concerned with initializing some files Kong creates on start, but I can't think of obvious reasons that would fail or be tied to an upgrade, as it's created fresh for each container replica. The chart changes related to that were to create a dedicated EmptyDir volume when we started using read-only root filesystems, but your Deployment isn't missing that and isn't trying to use some other prefix location, so I wouldn't expect any issues there.
You could file an upstream issue in https://github.com/Kong/kong to try and see if they can determine why the prefix initializer is reporting an error but no details. but if you are indeed able to clear the issue by starting from a fresh install that's probably quicker.
thanks, I already created an new issue: https://github.com/Kong/kong/issues/12418
And anybody knows why ingress-controller
container get this.
time="2024-01-12T10:51:41Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused" logger=setup retries=0/60
In order to avoid other effect, I uninstall others kong charts, only with this custom-values:
env:
database: "off"
admin_listen: "127.0.0.1:8001"
log_level: debug
admin:
enabled: "true"
http:
enabled: "true"
tls:
enabled: "false"
but looks like no change there
I found this issue: https://github.com/Kong/charts/issues/526, And set envs as @brennoo said, but it's useless for me.
hey @NICK-DUAN,
It seems you need to set admin_listen
port to 8444, check the admin port here https://github.com/Kong/charts/issues/983#issuecomment-1899906354
hey @NICK-DUAN,
It seems you need to set
admin_listen
port to 8444, check the admin port here #983 (comment)
I install kong/kong
, and I get a service that port is 8001
, it seems right
# Source: kong/templates/service-kong-admin.yaml
apiVersion: v1
kind: Service
metadata:
name: kong-kong-admin
namespace: kong
labels:
app.kubernetes.io/name: kong
helm.sh/chart: kong-2.33.3
app.kubernetes.io/instance: "kong"
app.kubernetes.io/managed-by: "Helm"
app.kubernetes.io/version: "3.5"
spec:
type: NodePort
ports:
- name: kong-admin
port: 8001
targetPort: 8001
protocol: TCP
selector:
app.kubernetes.io/name: kong
app.kubernetes.io/component: app
app.kubernetes.io/instance: "kong"
And I have try this: https://github.com/Kong/charts/issues/983#issuecomment-1909273039, with this custom-values:
env:
database: "off"
admin_listen: "127.0.0.1:8444 ssl"
log_level: debug
ingressController:
env:
kong_admin_init_retries: 5
kong_admin_init_retry_delay: "20s"
admin:
enabled: true
http:
ebabled: false
tls:
ebabled: true
change the admin port and still retry and resused..., it doesn't help me.
time="2024-01-25T11:51:08Z" level=info msg="diagnostics server disabled"
time="2024-01-25T11:51:08Z" level=info msg="starting controller manager" commit=956f457aafc2a07910bfa3c496e549493940e1e1 logger=setup release=2.12.3 repo="https://github.com/Kong/kubernetes-ingress-controller.git"
time="2024-01-25T11:51:08Z" level=info msg="the ingress class name has been set" logger=setup value=kong
time="2024-01-25T11:51:08Z" level=info msg="getting enabled options and features" logger=setup
time="2024-01-25T11:51:08Z" level=info msg="getting the kubernetes client configuration" logger=setup
W0125 11:51:08.496959 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2024-01-25T11:51:08Z" level=info msg="starting standalone health check server" logger=setup
time="2024-01-25T11:51:08Z" level=info msg="getting the kong admin api client configuration" logger=setup
time="2024-01-25T11:51:08Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=0/5
time="2024-01-25T11:51:28Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=1/5
time="2024-01-25T11:51:48Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=2/5
time="2024-01-25T11:52:08Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=3/5
time="2024-01-25T11:52:28Z" level=info msg="Retrying kong admin api client call after error" error="making HTTP request: Get \"https://localhost:8444/\": dial tcp [::1]:8444: connect: connection refused" logger=setup retries=4/5
Error: could not retrieve Kong admin root(s): making HTTP request: Get "https://localhost:8444/": dial tcp [::1]:8444: connect: connection refused
new custom-values.yaml
proxy:
type: ClusterIP
env:
admin_listen: "127.0.0.1:8444 ssl"
ingressController:
env:
kong_admin_init_retries: 5
kong_admin_init_retry_delay: "20s"
ingressClass: "testkong"
admin:
enabled: true
tls:
enabled: true
type: ClusterIP
containerSecurityContext: {}
replicaCount: 1
manager:
enabled: false
portal:
enabled: false
portalapi:
enabled: false
cluster:
enabled: false
clustertelemetry:
enabled: false
then I delete securityContext, it looks a little better, but another problem is comming:
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.IngressClassParameters worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.Dynamic/HTTPRoute worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.Dynamic/KnativeV1Alpha1/Ingress worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.KongConsumerGroup worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.KongConsumer worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.Dynamic/Gateway worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.KongClusterPlugin worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.IngressClass.netv1 worker count=1
time="2024-02-05T02:53:07Z" level=info msg="Starting workers" logger=controllers.Ingress.netv1 worker count=1
time="2024-02-05T02:53:14Z" level=info msg="successfully synced configuration to Kong" update_strategy=InMemory url="https://localhost:8444"
[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
> goroutine 144 [running]:
> runtime/debug.Stack()
> /usr/local/go/src/runtime/debug/stack.go:24 +0x5e
> sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/log.go:60 +0xcd
> sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error(0xc000135a00, {0x2194ec0, 0xc0028f51e0}, {0x1ed57a4, 0x21}, {0x0, 0x0, 0x0})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/log/deleg.go:139 +0x5d
> github.com/go-logr/logr.Logger.Error({{0x21b5f28?, 0xc000135a00?}, 0x0?}, {0x2194ec0, 0xc0028f51e0}, {0x1ed57a4, 0x21}, {0x0, 0x0, 0x0})
> /go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:299 +0xda
> sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1.1({0x21b2590?, 0xc0008e7a40?})
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/source/kind.go:68 +0x1a9
> k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func2(0xc00281de00?, {0x21b2590?, 0xc0008e7a40?})
> /go/pkg/mod/k8s.io/apimachinery@v0.28.2/pkg/util/wait/loop.go:73 +0x52
> k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x21b2590, 0xc0008e7a40}, {0x21a8ef0?, 0xc0008e9040}, 0x1, 0x0, 0x0?)
> /go/pkg/mod/k8s.io/apimachinery@v0.28.2/pkg/util/wait/loop.go:74 +0x233
> k8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel({0x21b2590, 0xc0008e7a40}, 0x0?, 0x0?, 0x0?)
> /go/pkg/mod/k8s.io/apimachinery@v0.28.2/pkg/util/wait/poll.go:33 +0x56
> sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start.func1()
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/source/kind.go:56 +0xee
> created by sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind).Start in goroutine 139
> /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/source/kind.go:48 +0x1d8
time="2024-02-05T02:55:06Z" level=error msg="Could not wait for Cache to sync" error="failed to wait for DiscoveryV1EndpointSlice caches to sync: timed out waiting for cache to be synced for Kind *v1.EndpointSlice" logger=controllers.EndpointSlice
time="2024-02-05T02:55:06Z" level=info msg="Stopping and waiting for non leader election runnables"
time="2024-02-05T02:55:06Z" level=info msg="Stopping and waiting for leader election runnables"
time="2024-02-05T02:55:06Z" level=info msg="context done: shutting down the proxy update server" subsystem=dataplane-synchronizer
time="2024-02-05T02:55:06Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Ingress.netv1
time="2024-02-05T02:55:06Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.IngressClass.netv1
time="2024-02-05T02:55:06Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.KongClusterPlugin
time="2024-02-05T02:55:06Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.Dynamic/Gateway
time="2024-02-05T02:55:06Z" level=info msg="Shutdown signal received, waiting for all workers to finish" logger=controllers.KongConsumer
now proxy
container running well, but ingress-controller
container restart all the time. anybody help?
You likely need to update CRDs: https://github.com/Kong/charts/blob/main/charts/kong/UPGRADE.md#updates-to-crds
The controller framework eventually aborts if you ask it to manage resources it cannot access, so missing CRDs or missing permissions will cause that sort of failure. Helm will keep the permissions updated so long as you use a chart version released after your KIC version, but it does not update CRDs.
@NICK-DUAN
To config the admin listening address. you should update this config instead of env
.
admin:
enabled: true
addresses:
- 127.0.0.1
Clearly, here is my helm diff after adding the above code. And this solve my issue on both containers.
@NICK-DUAN To config the admin listening address. you should update this config instead of
env
.admin: enabled: true addresses: - 127.0.0.1
Clearly, here is my helm diff after adding the above code. And this solve my issue on both containers.
Thanks, I'll try
latest progress:
got ingress-controller report logs:
2024-05-17T09:27:41Z info Diagnostics server disabled {"v": 0}
2024-05-17T09:27:41Z info setup Starting controller manager {"v": 0, "release": "3.0.3", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "cbd7866dbd449fd32b1c9f56cf747e68b577973d"}
2024-05-17T09:27:41Z info setup The ingress class name has been set {"v": 0, "value": "kong-test"}
2024-05-17T09:27:41Z info setup Getting enabled options and features {"v": 0}
2024-05-17T09:27:41Z info setup Getting the kubernetes client configuration {"v": 0}
W0517 09:27:41.877939 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2024-05-17T09:27:41Z info setup Starting standalone health check server {"v": 0}
2024-05-17T09:27:41Z info setup Getting the kong admin api client configuration {"v": 0}
2024-05-17T09:27:41Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "0/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:42Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "1/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:43Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "2/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:44Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "3/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:45Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "4/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:46Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "5/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:47Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "6/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:48Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "7/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:49Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "8/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:50Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "9/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-05-17T09:27:51Z info setup Retrying kong admin api client call after error {"v": 0, "retries": "10/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
Then I thought that the proxy container exposed port 8444, bug proxy report logs
Error:
Run with --v (verbose) or --vv (debug) for more details
Then I change proxy command
and args
:
command:
- ./docker-entrypoint.sh
args:
- kong
- docker-start
- --vv
And I got logs:
Error:
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28:
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:28: in function 'load_conf'
/usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:85: in function </usr/local/share/lua/5.1/kong/cmd/utils/inject_confs.lua:84>
[C]: in function 'xpcall'
/usr/local/bin/kong:99: in function 'file_gen'
init_worker_by_lua(nginx.conf:193):51: in function <init_worker_by_lua(nginx.conf:193):49>
[C]: in function 'xpcall'
init_worker_by_lua(nginx.conf:193):58: in function <init_worker_by_lua(nginx.conf:193):56>
So I think the problem is that the proxy container is not running properly. but I don't know how to fix it?
delete container securityContext, proxy got logs:
2024/05/17 10:24:56 [notice] 1281#0: *308 [lua] ready.lua:111: fn(): not ready for proxying: no configuration available (empty configuration present), client: 9.149.180.162, server: kong_status, request: "GET /status/ready HTTP/1.1", host: "9.149.182.119:8100"
2024/05/17 10:24:56 [debug] 1281#0: *308 [lua] init.lua:23: poll(): worker-events: emulate poll method
2024/05/17 10:24:58 [debug] 1281#0: *309 [lua] init.lua:23: poll(): worker-events: emulate poll method
2024/05/17 10:25:03 [debug] 1281#0: *303 [lua] init.lua:23: poll(): worker-events: emulate poll method
--set admin.enabled=true
Solved the issue for me, btw.
Removing the following configuration can resolve the issue:
seccompProfile:
type: RuntimeDefault
when i upgrade
kong/charts
(from 1.7.0 to 2.30.0),kong
andingress-controller
not running.my values.yaml
ingress-controller
log:proxy
log:My Kong was upgraded from a very old version, and I don’t know what changes happened in the meantime. anybody help?