search

kubeedge / kubeedge

Kubernetes Native Edge Computing Framework (project under CNCF)
https://kubeedge.io
Apache License 2.0
6.49k stars 1.67k forks source link

Service Account failed to be authenticated on the secondly added edge node with requireAuthorization feature gate enabled #5606

Open IterableTrucks opened 2 weeks ago

IterableTrucks commented 2 weeks ago

What happened: Pods on the first added edge node run normally with requireAuthorization feature gate enabled. But after I add the second edge node with the same edgecore configuration, the pod running on the second edge node cannot request k8s api： authentication.go:73] "Unable to authenticatethe request" err="serviceaccount ns1/sa1 not found". Meanwhile the pod with same manifest runs normally on the first edge node.

What you expected to happen: Pods can request k8s api on every edge node.

How to reproduce it (as minimally and precisely as possible):

The configuration of edgecore: ```yaml apiVersion: edgecore.config.kubeedge.io/v1alpha2 database: aliasName: default dataSource: /var/lib/kubeedge/edgecore.db driverName: sqlite3 kind: EdgeCore featureGates: requireAuthorization: true modules: dbTest: enable: false deviceTwin: dmiSockPath: /etc/kubeedge/dmi.sock enable: true edgeHub: enable: true heartbeat: 15 httpServer: https://192.168.3.45:10002 messageBurst: 60 messageQPS: 30 projectID: e632aba927ea4ac2b575ec1603d56f10 quic: enable: false handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10001 writeDeadline: 15 rotateCertificates: true tlsCaFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/server.crt tlsPrivateKeyFile: /etc/kubeedge/certs/server.key token: "" websocket: enable: true handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10000 writeDeadline: 15 edgeStream: enable: true handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10004 tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt tlsTunnelCertFile: /etc/kubeedge/certs/server.crt tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key writeDeadline: 15 edged: containerRuntime: remote enable: true hostnameOverride: nm178#(nm177 on the first node) masterServiceNamespace: default maxContainerCount: -1 maxPerPodContainerCount: 1 minimumGCAge: 0s podSandboxImage: kubeedge/pause:3.6 registerNode: true registerNodeNamespace: default registerSchedulable: true remoteImageEndpoint: unix:///run/containerd/containerd.sock remoteRuntimeEndpoint: unix:///run/containerd/containerd.sock rootDirectory: /var/lib/edged tailoredKubeletConfig: address: 127.0.0.1 cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 169.254.96.16 clusterDomain: cluster.local configMapAndSecretChangeDetectionStrategy: Get containerLogMaxFiles: 5 containerLogMaxSize: 10Mi containerRuntimeEndpoint: unix:///var/run/crio/crio.sock contentType: application/json cpuCFSQuota: true cpuCFSQuotaPeriod: 100ms cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebugFlagsHandler: true enableDebuggingHandlers: true enableProfilingHandler: true enableSystemLogHandler: true enforceNodeAllocatable: - pods eventBurst: 100 eventRecordQPS: 50 evictionHard: imagefs.available: 5% memory.available: 100Mi nodefs.available: 3% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: false fileCheckFrequency: 20s hairpinMode: promiscuous-bridge imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s imageServiceEndpoint: unix:///var/run/crio/crio.sock iptablesDropBit: 15 iptablesMasqueradeBit: 14 localStorageCapacityIsolation: true logging: flushFrequency: 5s format: text options: json: infoBufferSize: "0" verbosity: 0 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 memoryManagerPolicy: None memorySwap: {} memoryThrottlingFactor: 0.9 nodeLeaseDurationSeconds: 40 nodeStatusMaxImages: 0 nodeStatusReportFrequency: 5m0s nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 readOnlyPort: 10350 registerNode: true registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf runtimeRequestTimeout: 2m0s seccompDefault: false serializeImagePulls: true shutdownGracePeriod: 0s shutdownGracePeriodCriticalPods: 0s staticPodPath: /etc/kubeedge/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s topologyManagerPolicy: none topologyManagerScope: container volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ volumeStatsAggPeriod: 1m0s eventBus: enable: true eventBusTLS: enable: false tlsMqttCAFile: /etc/kubeedge/ca/rootCA.crt tlsMqttCertFile: /etc/kubeedge/certs/server.crt tlsMqttPrivateKeyFile: /etc/kubeedge/certs/server.key mqttMode: 0 mqttPassword: "" mqttPubClientID: "" mqttQOS: 0 mqttRetain: false mqttServerExternal: tcp://127.0.0.1:1883 mqttServerInternal: tcp://127.0.0.1:1884 mqttSessionQueueSize: 100 mqttSubClientID: "" mqttUsername: "" metaManager: contextSendGroup: hub contextSendModule: websocket enable: true metaServer: apiAudiences: null dummyServer: 169.254.30.10:10550 enable: true server: 127.0.0.1:10550 serviceAccountIssuers: - https://kubernetes.default.svc.cluster.local serviceAccountKeyFiles: null tlsCaFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/server.crt tlsPrivateKeyFile: /etc/kubeedge/certs/server.key remoteQueryTimeout: 60 serviceBus: enable: false port: 9060 server: 127.0.0.1 timeout: 60 ```

Anything else we need to know?:

Environment:

Shelley-BaoYue commented 1 week ago

I will try to reproduce it in my own environment and it may take a while. If you have any new progress, please feel free to communicate here.

zhuyaguang commented 1 week ago

my edgecore logs also Appear logs

May 21 00:15:56 edge1 edgecore[6586]: E0521 00:15:56.484334 6586 authentication.go:73] "Unable to authenticate the request" err="tokenData not found when authenticating"

Shelley-BaoYue commented 1 week ago

my edgecore logs also Appear logs

May 21 00:15:56 edge1 edgecore[6586]: E0521 00:15:56.484334 6586 authentication.go:73] "Unable to authenticate the request" err="tokenData not found when authenticating"

Does the problem also occur when multiple edge nodes are connected?