search

kubeedge / kubeedge

Kubernetes Native Edge Computing Framework (project under CNCF)
https://kubeedge.io
Apache License 2.0
6.52k stars 1.68k forks source link

想使用 v1.17.0 InClusterConfig 功能 需要进行额外的配置么？默认配置还是报错KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined #5608

Closed thinkeng closed 1 month ago

thinkeng commented 1 month ago

k8s v1.27.2 + docker v24.0.7 + cri-docker v0.3.10 + edgecloud v1.17.0

edgecore v1.17.0 + dokcer v26.0.0 + cri-docker v0.3.12

在 云端 cloudcore cm 中，然后重启pod

.......
dynamicController:
     enable: true
     requireAuthorization: true   //增加这条配置
edgeController:

在边缘端 

..........
metaManager:
    contextSendGroup: hub
    contextSendModule: websocket
    enable: true
    metaServer:
      requireAuthorization: true  // 增加这条配置
      apiAudiences: null
.....................

但是日志还是报下面错误,是上面配置不对么

W0513 13:13:55.799080       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0513 13:13:55.799111       1 client_config.go:613] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
{"error":"invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","level":"fatal","msg":"error building kubernetes config","source":"k8s/k8s.go:21","time":"2024-05-13T13:13:55Z"}
Shelley-BaoYue commented 1 month ago

refer to https://github.com/kubeedge/kubeedge/issues/5586#issuecomment-2102583163 to set featureGates 😄

thinkeng commented 1 month ago

refer to #5586 (comment) to set featureGates 😄

配置后部署 需要加载in-cluster conf 的时候（部署https://github.com/4paradigm/k8s-vgpu-scheduler ）， 会报下面的错误，然后 边缘节点的 node 会 NotReady 状态，cloudcore 所在的node 上没有部署 edgemesh-agent

cloudcore 的错误日志如下：

I0515 10:58:22.649222       1 node_session.go:137] Start session for edge node barry-edge-aibox-01
I0515 10:58:22.722992       1 upstream.go:89] Dispatch message: cebc4894-ae3b-480b-ae39-267c880de6f8
I0515 10:58:22.723018       1 upstream.go:96] Message: cebc4894-ae3b-480b-ae39-267c880de6f8, resource type is: membership/detail
W0515 10:58:23.870590       1 reflector.go:535] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "serviceaccounts" in API group "" at the cluster scope
E0515 10:58:23.870627       1 reflector.go:147] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "serviceaccounts" in API group "" at the cluster scope
I0515 10:58:24.581280       1 tunnelserver.go:121] get a new tunnel agent hostname barry-edge-aibox-01, internalIP 192.168.8.19
W0515 10:58:24.867957       1 reflector.go:535] k8s.io/client-go/informers/factory.go:150: failed to list *v1.CertificateSigningRequest: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:24.867990       1 reflector.go:147] k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CertificateSigningRequest: failed to list *v1.CertificateSigningRequest: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:26.406436       1 upstream.go:1044] message: 6a575fcb-05c4-4074-9d0a-5ca031d5ec5c process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
E0515 10:58:27.910389       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:30.280279       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:34.911303       1 upstream.go:777] apiserver get service account token failed: err pods "edgemesh-agent-wl9g8" not found
W0515 10:58:34.911328       1 upstream.go:703] message: 7ded5928-791d-4788-b54a-b45f0a1ef701 process failure, resource not found, namespace: kubeedge, name: edgemesh-agent
E0515 10:58:35.068586       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:36.405166       1 upstream.go:1044] message: c2da383b-18af-465d-bcd8-d589f0a2581f process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
W0515 10:58:39.077347       1 reflector.go:535] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: failed to list *v1alpha1.ServiceAccountAccess: serviceaccountaccesses.policy.kubeedge.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "serviceaccountaccesses" in API group "policy.kubeedge.io" at the cluster scope
E0515 10:58:39.077378       1 reflector.go:147] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: Failed to watch *v1alpha1.ServiceAccountAccess: failed to list *v1alpha1.ServiceAccountAccess: serviceaccountaccesses.policy.kubeedge.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "serviceaccountaccesses" in API group "policy.kubeedge.io" at the cluster scope
E0515 10:58:43.387824       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
E0515 10:58:46.410428       1 upstream.go:1044] message: 75633096-c873-4aee-98c9-479419a3aa42 process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
W0515 10:58:50.306539       1 reflector.go:535] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "roles" in API group "rbac.authorization.k8s.io" at the cluster scope
E0515 10:58:50.306570       1 reflector.go:147] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "roles" in API group "rbac.authorization.k8s.io" at the cluster scope
E0515 10:58:56.406373       1 upstream.go:1044] message: 630c285f-193f-4aab-962e-4c6cfd0b0fb0 process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8

W0515 10:58:58.970368       1 reflector.go:535] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: failed to list *v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E0515 10:58:58.970400       1 reflector.go:147] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: Failed to watch *v1.RoleBinding: failed to list *v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

E0515 10:59:00.414457       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
^[cW0515 10:59:01.018243       1 reflector.go:535] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: failed to list *v1.ClusterRole: clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
E0515 10:59:01.018289       1 reflector.go:147] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:233: Failed to watch *v1.ClusterRole: failed to list *v1.ClusterRole: clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope

cloudcore-6bc8d4c566-tbvzq 0/1 CrashLoopBackOff

I0515 16:07:36.979456       1 upstream.go:96] Message: 3af40425-9743-4000-b9a6-a013aa92a2b2, resource type is: membership/detail
I0515 16:07:36.979461       1 upstream.go:89] Dispatch message: 3addb571-0f5d-40f1-beb4-92989cfb7a48
I0515 16:07:36.979467       1 upstream.go:96] Message: 3addb571-0f5d-40f1-beb4-92989cfb7a48, resource type is: membership/detail
I0515 16:07:37.018290       1 upstream.go:89] Dispatch message: 2bc3c3d3-626e-4a66-8c2d-8e027216b53e
I0515 16:07:37.018308       1 upstream.go:96] Message: 2bc3c3d3-626e-4a66-8c2d-8e027216b53e, resource type is: membership/detail
I0515 16:07:37.033522       1 upstream.go:89] Dispatch message: cf19ed63-989c-4114-8bba-d38eab084f2c
I0515 16:07:37.033540       1 upstream.go:96] Message: cf19ed63-989c-4114-8bba-d38eab084f2c, resource type is: membership/detail
E0515 16:07:37.035048       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
I0515 16:07:37.067514       1 upstream.go:89] Dispatch message: b5238eaa-e8f6-4f7a-92d5-0728f0602298
I0515 16:07:37.067547       1 upstream.go:96] Message: b5238eaa-e8f6-4f7a-92d5-0728f0602298, resource type is: membership/detail
I0515 16:07:37.100398       1 upstream.go:89] Dispatch message: 8bccea99-f180-45af-abda-b1215cb5f049
I0515 16:07:37.100418       1 upstream.go:96] Message: 8bccea99-f180-45af-abda-b1215cb5f049, resource type is: membership/detail
[controller-runtime] log.SetLogger(...) was never called, logs will not be displayed:
goroutine 3245 [running]:
runtime/debug.Stack()
    /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:59 +0xbd
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Error(0xc00088aa40, {0x28be440, 0xc0030fa620}, {0x25742d4, 0x20}, {0x0, 0x0, 0x0})
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/log/deleg.go:139 +0x68
github.com/kubeedge/kubeedge/vendor/github.com/go-logr/logr.Logger.Error({{0x28eb578?, 0xc00088aa40?}, 0x4442b1?}, {0x28be440, 0xc0030fa620}, {0x25742d4, 0x20}, {0x0, 0x0, 0x0})
    /go/src/github.com/kubeedge/kubeedge/vendor/github.com/go-logr/logr/logr.go:299 +0xda
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.1({0x28e4d70?, 0xc00049d360?}, 0xc0003d2320, {0x28d07a8, 0xc0004accf0})
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202 +0x186
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2(0xc0003d2320, {0x28e4d70?, 0xc00049d360}, 0xc00064e3a0)
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:207 +0x418
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start(0xc0003d2320, {0x28e4d70, 0xc00049d360})
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:233 +0x165
github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0xc000618c60)
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:219 +0xdb
created by github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile
    /go/src/github.com/kubeedge/kubeedge/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:203 +0x1ad
E0515 16:07:46.362762       1 upstream.go:1044] message: 7591143f-0ac4-43b5-968e-558f3f4f2298 process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
E0515 16:07:54.639966       1 upstream.go:777] apiserver get service account token failed: err pods "edgemesh-agent-wl9g8" not found
W0515 16:07:54.639992       1 upstream.go:703] message: a1635f88-95e4-4d07-9d74-203669e0b600 process failure, resource not found, namespace: kubeedge, name: edgemesh-agent
E0515 16:07:56.362200       1 upstream.go:1044] message: 0728eb57-c08f-464a-9c6f-459371eab014 process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
E0515 16:08:06.361879       1 upstream.go:1044] message: fdc3dd7e-2742-45fe-bd62-c979e2974e8e process failure, patch pod failed with error: pods "edgemesh-agent-wl9g8" not found, namespace: kubeedge, name: edgemesh-agent-wl9g8
E0515 16:08:09.066397       1 upstream.go:1327] create CertificateSigningRequests metaserver-csr-barry-edge-aibox-01 failed, error: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:serviceaccount:kubeedge:cloudcore" cannot create resource "certificatesigningrequests" in API group "certificates.k8s.io" at the cluster scope
F0515 16:08:09.134946       1 policycontroller.go:102] failed to start controller manager, [failed to wait for serviceaccountaccess caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.ServiceAccountAccess, failed waiting for all runnables to end within grace period of 30s: context deadline exceeded]
Shelley-BaoYue commented 1 month ago

CSR related clusterRoleBinding will be created when you using keadm init cloudcore and set cloudCore.featureGates.requireAuthorization=true. If you config the featureGates and then restart cloudcore, thie clusterRoleBinding will not be created and you need to create it munually refer to https://github.com/kubeedge/kubeedge/blob/master/manifests/charts/cloudcore/templates/rbac_cloudcore_feature.yaml

thinkeng commented 1 month ago

CSR related clusterRoleBinding will be created when you using keadm init cloudcore and set cloudCore.featureGates.requireAuthorization=true. If you config the featureGates and then restart cloudcore, thie clusterRoleBinding will not be created and you need to create it munually refer to https://github.com/kubeedge/kubeedge/blob/master/manifests/charts/cloudcore/templates/rbac_cloudcore_feature.yaml

可以了，谢谢