WillardHu
commented
1 month ago cc @Shelley-BaoYue
Closed IterableTrucks closed 1 month ago
WillardHu
commented
1 month ago cc @Shelley-BaoYue
Shelley-BaoYue
commented
1 month ago Have you ever enable metaserver and dynamiccontroller and set featureGates requireAuthorization=true both in cloudcore and edgecore yaml？ https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG/CHANGELOG-1.17.md#important-steps-before-upgrading
IterableTrucks
commented
1 month ago dynamicController and requireAuthorization are enabled on cloudcore and metaServer is enabled on edgecore.
IterableTrucks
commented
1 month ago Have you ever enable metaserver and dynamiccontroller and set featureGates requireAuthorization=true both in cloudcore and edgecore yaml？ https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG/CHANGELOG-1.17.md#important-steps-before-upgrading
Do you mean the metaServer option should be enabled on cloudcore and dynamicController & requireAuthorization feature gate should be enabled on edgecore? But I cannot find these options in helm chart of cloudcore and edgecore.yaml of edgecore
Shelley-BaoYue
commented
1 month ago apiVersion: edgecore.config.kubeedge.io/v1alpha1
kind: EdgeCore
featureGates:
requireAuthorization: true
...apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
featureGates:
requireAuthorization: true
...featureGates option is supported in helm chart of cloudcore now with .Values.cloudCore.featureGates.requireAuthorization https://github.com/kubeedge/kubeedge/blob/master/manifests/charts/cloudcore/templates/configmap_cloudcore.yaml#L13; and is not supported in edgecore.yaml now, you should modify the edgecore.yaml and restart edgecore.
IterableTrucks
commented
1 month ago
- in edgecore config: enable metaServer; set requireAuthorization=true
apiVersion: edgecore.config.kubeedge.io/v1alpha1 kind: EdgeCore featureGates: requireAuthorization: true ...
- in cloudcore config: enable dynamiccontroller; set requireAuthorization=true
apiVersion: cloudcore.config.kubeedge.io/v1alpha1 kind: CloudCore featureGates: requireAuthorization: true ...featureGates option is supported in helm chart of cloudcore now with
.Values.cloudCore.featureGates.requireAuthorizationhttps://github.com/kubeedge/kubeedge/blob/master/manifests/charts/cloudcore/templates/configmap_cloudcore.yaml#L13; and is not supported in edgecore.yaml now, you should modify the edgecore.yaml and restart edgecore.
The edgecore failed to restart after I configured featureGates.requireAuthorization=true in edgecore.yaml
Shelley-BaoYue
commented
1 month ago Maybe somethink unusual about your kernel? Could you please try ip link add test type dummy and see if you are able to create dummy interface manually?
IterableTrucks
commented
1 month ago Maybe somethink unusual about your kernel? Could you please try
ip link add test type dummyand see if you are able to create dummy interface manually?
You are right. Everything is OK after I recompile the kernel with CONFIG_DUMMY=y and reboot the edge node with the new kernel.
What happened: After uprading kubeedge on both cloud side and edge side to v1.17.0, the pod on edge nodes still doesn't have
KUBERNETES_SERVICE_HOSTandKUBERNETES_SERVICE_PORTconfigured What you expected to happen: The two environment variables are configured in pods. How to reproduce it (as minimally and precisely as possible):The cloudcore configmap is:
```yaml apiVersion: cloudcore.config.kubeedge.io/v1alpha2 kind: CloudCore featureGates: requireAuthorization: true kubeAPIConfig: kubeConfig: "" master: "" modules: cloudHub: advertiseAddress: - 192.168.3.45 dnsNames: - nodeLimit: 1000 tlsCAFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/edge.crt tlsPrivateKeyFile: /etc/kubeedge/certs/edge.key unixsocket: address: unix:///var/lib/kubeedge/kubeedge.sock enable: true websocket: address: 0.0.0.0 enable: true port: 10000 quic: address: 0.0.0.0 enable: false maxIncomingStreams: 10000 port: 10001 https: address: 0.0.0.0 enable: true port: 10002 cloudStream: enable: true streamPort: 10003 tunnelPort: 10004 dynamicController: enable: true router: enable: true iptablesManager: enable: true mode: internal taskManager: enable: true ``` The requireAuthorization feature gate and dynamicController are enabled as noted in [changelog ](https://github.com/kubeedge/kubeedge/blob/master/CHANGELOG/CHANGELOG-1.17.md)and '192.168.3.45' is the IP of cloud side node.The edgecore config on edge node:
```yaml apiVersion: edgecore.config.kubeedge.io/v1alpha2 database: aliasName: default dataSource: /var/lib/kubeedge/edgecore.db driverName: sqlite3 kind: EdgeCore modules: dbTest: enable: false deviceTwin: dmiSockPath: /etc/kubeedge/dmi.sock enable: true edgeHub: enable: true heartbeat: 15 httpServer: https://192.168.3.45:10002 messageBurst: 60 messageQPS: 30 projectID: e632aba927ea4ac2b575ec1603d56f10 quic: enable: false handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10001 writeDeadline: 15 rotateCertificates: true tlsCaFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/server.crt tlsPrivateKeyFile: /etc/kubeedge/certs/server.key token: "" websocket: enable: true handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10000 writeDeadline: 15 edgeStream: enable: true handshakeTimeout: 30 readDeadline: 15 server: 192.168.3.45:10004 tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt tlsTunnelCertFile: /etc/kubeedge/certs/server.crt tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key writeDeadline: 15 edged: containerRuntime: remote enable: true hostnameOverride: edge01 masterServiceNamespace: default maxContainerCount: -1 maxPerPodContainerCount: 1 minimumGCAge: 0s podSandboxImage: kubeedge/pause:3.6 registerNode: true registerNodeNamespace: default registerSchedulable: true remoteImageEndpoint: unix:///run/containerd/containerd.sock remoteRuntimeEndpoint: unix:///run/containerd/containerd.sock rootDirectory: /var/lib/edged tailoredKubeletConfig: address: 127.0.0.1 cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 169.254.96.16 clusterDomain: cluster.local configMapAndSecretChangeDetectionStrategy: Get containerLogMaxFiles: 5 containerLogMaxSize: 10Mi containerRuntimeEndpoint: unix:///var/run/crio/crio.sock contentType: application/json cpuCFSQuota: true cpuCFSQuotaPeriod: 100ms cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebugFlagsHandler: true enableDebuggingHandlers: true enableProfilingHandler: true enableSystemLogHandler: true enforceNodeAllocatable: - pods eventBurst: 100 eventRecordQPS: 50 evictionHard: imagefs.available: 5% memory.available: 100Mi nodefs.available: 3% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: false fileCheckFrequency: 20s hairpinMode: promiscuous-bridge imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s imageServiceEndpoint: unix:///var/run/crio/crio.sock iptablesDropBit: 15 iptablesMasqueradeBit: 14 localStorageCapacityIsolation: true logging: flushFrequency: 5s format: text options: json: infoBufferSize: "0" verbosity: 0 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 memoryManagerPolicy: None memorySwap: {} memoryThrottlingFactor: 0.9 nodeLeaseDurationSeconds: 40 nodeStatusMaxImages: 0 nodeStatusReportFrequency: 5m0s nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 readOnlyPort: 10350 registerNode: true registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf runtimeRequestTimeout: 2m0s seccompDefault: false serializeImagePulls: true shutdownGracePeriod: 0s shutdownGracePeriodCriticalPods: 0s staticPodPath: /etc/kubeedge/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s topologyManagerPolicy: none topologyManagerScope: container volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ volumeStatsAggPeriod: 1m0s eventBus: enable: true eventBusTLS: enable: false tlsMqttCAFile: /etc/kubeedge/ca/rootCA.crt tlsMqttCertFile: /etc/kubeedge/certs/server.crt tlsMqttPrivateKeyFile: /etc/kubeedge/certs/server.key mqttMode: 2 mqttPassword: "" mqttPubClientID: "" mqttQOS: 0 mqttRetain: false mqttServerExternal: tcp://192.168.6.176:1883 mqttServerInternal: tcp://127.0.0.1:1884 mqttSessionQueueSize: 100 mqttSubClientID: "" mqttUsername: "" metaManager: contextSendGroup: hub contextSendModule: websocket enable: true metaServer: apiAudiences: null dummyServer: 169.254.30.10:10550 enable: true server: 127.0.0.1:10550 serviceAccountIssuers: - https://kubernetes.default.svc.cluster.local serviceAccountKeyFiles: null tlsCaFile: /etc/kubeedge/ca/rootCA.crt tlsCertFile: /etc/kubeedge/certs/server.crt tlsPrivateKeyFile: /etc/kubeedge/certs/server.key remoteQueryTimeout: 60 serviceBus: enable: false port: 9060 server: 127.0.0.1 timeout: 60 ``` MetaServer is enabled and '192.168.6.176' is the IP of the edge node.Anything else we need to know?:
Environment:
kubectl version): v1.26.5cloudcore --versionandedgecore --version): v1.17.0Cloud nodes Environment:
- Hardware configuration (e.g. `lscpu`): x86_64 20cores 64GB RAM - OS (e.g. `cat /etc/os-release`): ubuntu 20.04.6 - Kernel (e.g. `uname -a`): 5.4.0-176-generic - Go version (e.g. `go version`): - Others:Edge nodes Environment:
- edgecore version (e.g. `edgecore --version`):v1.17.0 - Hardware configuration (e.g. `lscpu`): aarch64 8cores 6GB RAM - OS (e.g. `cat /etc/os-release`): Ubuntu 20.04.5 - Kernel (e.g. `uname -a`): 4.14.48 - Go version (e.g. `go version`): - Others: