search

apache / rocketmq-operator

Apache RocketMQ Operator
https://rocketmq.apache.org/
Apache License 2.0
309 stars 126 forks source link

deploy failed：Failed to update Broker Size status. #141

Closed chaoyoung closed 1 year ago

chaoyoung commented 1 year ago

BUG REPORT

Please describe the issue you observed:

What did you do (The steps to reproduce)?

What did you expect to see?

What did you see instead?

Please tell us about your environment:

k8s version: 1.24.6 rocketmq-operator version: 0.4.0 rocketmq version: 5.0.0

Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

console pod not found and pod rocketmq-operator-5b59b6f6f8-gzkvw print the following errors:

2022-11-09T10:06:32.854Z    ERROR   controller_broker   Failed to update Broker Size status.    {"Request.Namespace": "default", "Request.Name": "broker", "error": "Broker.rocketmq.apache.org \"broker\" is invalid: status.nodes: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:227

pod broker-0-master-0 print the following errors:

2022-11-09T10:10:55.736Z    INFO    controller.dledger-controller   Starting Controller
E1109 10:10:55.758673       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:10:55.758673       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:55.758927       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:55.759326       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:55.759879       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:55.759873       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:56.579821       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:56.579898       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:57.059921       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:57.193971       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:57.223319       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:10:57.225168       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:58.368262       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:58.603102       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:58.844578       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:59.589131       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:10:59.655864       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:10:59.687660       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:02.757662       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:03.070316       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:03.219899       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:03.464882       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:04.909663       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:04.930091       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:11:10.833216       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:10.842801       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:11.890116       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:14.638949       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:11:15.285896       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:15.288424       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:25.294127       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:28.488022       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:11:31.712048       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:33.529766       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:36.213991       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:37.799082       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:11:59.078102       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:04.929693       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:11.438412       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:14.864584       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:16.304905       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:12:17.129334       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:51.078080       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:52.045342       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:12:54.546082       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
2022-11-09T10:12:55.736Z    ERROR   controller.broker-controller    Could not wait for Cache to sync    {"error": "failed to wait for broker-controller caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2022-11-09T10:12:55.737Z    ERROR   controller.dledger-controller   Could not wait for Cache to sync    {"error": "failed to wait for dledger-controller caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2022-11-09T10:12:55.736Z    ERROR   controller.console-controller   Could not wait for Cache to sync    {"error": "failed to wait for console-controller caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2022-11-09T10:12:55.739Z    ERROR   controller.topictransfer-controller Could not wait for Cache to sync    {"error": "failed to wait for topictransfer-controller caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2022-11-09T10:12:55.739Z    ERROR   error received after stop sequence was engaged  {"error": "failed to wait for dledger-controller caches to sync: timed out waiting for cache to be synced"}
2022-11-09T10:12:55.739Z    ERROR   error received after stop sequence was engaged  {"error": "failed to wait for console-controller caches to sync: timed out waiting for cache to be synced"}
2022-11-09T10:12:55.739Z    ERROR   error received after stop sequence was engaged  {"error": "failed to wait for topictransfer-controller caches to sync: timed out waiting for cache to be synced"}
2022-11-09T10:12:55.737Z    ERROR   controller.nameservice-controller   Could not wait for Cache to sync    {"error": "failed to wait for nameservice-controller caches to sync: timed out waiting for cache to be synced"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:234
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/manager/internal.go:696
2022-11-09T10:12:55.739Z    ERROR   setup   problem running manager {"error": "failed to wait for broker-controller caches to sync: timed out waiting for cache to be synced"}

pod name-service-0 print the following errors:

2022-11-09T10:15:59.780Z    INFO    controller.nameservice-controller   Starting Controller
E1109 10:15:59.861916       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:15:59.862279       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:15:59.862418       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:15:59.862531       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:15:59.862726       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:15:59.864241       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:00.881710       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:00.971919       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:00.971983       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:16:01.039319       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:01.344203       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:01.345288       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:03.081156       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:03.257906       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:16:03.605365       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:03.654074       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:04.131873       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:04.194760       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:06.919461       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:07.690500       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:07.927694       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:08.238132       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:08.426132       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:09.000698       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:16:15.526031       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:16:15.806454       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:17.699264       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:17.893667       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:18.481795       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:19.610117       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:29.441079       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
E1109 10:16:32.296002       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Controller: failed to list *v1alpha1.Controller: controllers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "controllers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:36.327072       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.NameService: failed to list *v1alpha1.NameService: nameservices.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "nameservices" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:38.982457       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Console: failed to list *v1alpha1.Console: consoles.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "consoles" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:41.217319       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.Broker: failed to list *v1alpha1.Broker: brokers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "brokers" in API group "rocketmq.apache.org" at the cluster scope
E1109 10:16:41.544994       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.TopicTransfer: failed to list *v1alpha1.TopicTransfer: topictransfers.rocketmq.apache.org is forbidden: User "system:serviceaccount:default:default" cannot list resource "topictransfers" in API group "rocketmq.apache.org" at the cluster scope
caigy commented 1 year ago

It seems that something is wrong with RBAC. Please check if role.xml, role_binding.yaml and service_account.yaml are the latest version and successfully applied.

E1109 10:10:55.758673       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope
chaoyoung commented 1 year ago

It seems that something is wrong with RBAC. Please check if role.xml, role_binding.yaml and service_account.yaml are the latest version and successfully applied.

E1109 10:10:55.758673       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" at the cluster scope

name-server and broker are running after redeploy, but console pod not found, rocketmq-operator contains error log:

2022-11-10T04:07:05.204Z    ERROR   controller_broker   Failed to update Broker Size status.    {"Request.Namespace": "default", "Request.Name": "broker", "error": "Broker.rocketmq.apache.org \"broker\" is invalid: status.nodes: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:227
caigy commented 1 year ago

name-server and broker are running after redeploy, but console pod not found, rocketmq-operator contains error log:

Are there any error messages about rocketmq-console?

chaoyoung commented 1 year ago

name-server and broker are running after redeploy, but console pod not found, rocketmq-operator contains error log:

Are there any error messages about rocketmq-console?

It should be rocketmq-operator unable to recognize affinity field I added.

[root@dev-k8s-master1 rocketmq-operator]# k apply -f rocketmq_v1alpha1_rocketmq_cluster.yaml 
configmap/broker-config created
broker.rocketmq.apache.org/broker created
nameservice.rocketmq.apache.org/name-service created
error: error validating "rocketmq_v1alpha1_rocketmq_cluster.yaml": error validating data: ValidationError(Console.spec): unknown field "affinity" in org.apache.rocketmq.v1alpha1.Console.spec; if you choose to ignore these errors, turn validation off with --validate=false
chaoyoung commented 1 year ago

name-server and broker are running after redeploy, but console pod not found, rocketmq-operator contains error log:

Are there any error messages about rocketmq-console?

It should be rocketmq-operator unable to recognize affinity field I added.

[root@dev-k8s-master1 rocketmq-operator]# k apply -f rocketmq_v1alpha1_rocketmq_cluster.yaml 
configmap/broker-config created
broker.rocketmq.apache.org/broker created
nameservice.rocketmq.apache.org/name-service created
error: error validating "rocketmq_v1alpha1_rocketmq_cluster.yaml": error validating data: ValidationError(Console.spec): unknown field "affinity" in org.apache.rocketmq.v1alpha1.Console.spec; if you choose to ignore these errors, turn validation off with --validate=false

see https://github.com/apache/rocketmq-operator/issues/142