search

kubeedge / kubeedge

Kubernetes Native Edge Computing Framework (project under CNCF)
https://kubeedge.io
Apache License 2.0
6.76k stars 1.73k forks source link

curl metaserver with watch but no response #4699

Open neiba opened 1 year ago

neiba commented 1 year ago

What happened: curl metaserver with watch but no response

What you expected to happen: curl metaserver with watch, and get response

How to reproduce it (as minimally and precisely as possible):

  1. create cm-1
  2. create pod-1 and mount cm-1
  3. curl http://127.0.0.1:10550/api/v1/namespaces/default/configmaps?watch=ture

expected get cm-1, but get none.

Anything else we need to know?:

Environment:

Shelley-BaoYue commented 1 year ago

Sorry, I fail to reproduce the problem in my environment. Could you please provide more information including cloudcore.yaml, edgecore.yaml and the yaml of your cm, pod?

neiba commented 1 year ago
  1. create cm-1
  2. create pod-1 to edgenode, and pod mount cm-1
  3. ensure objectsync for cm-1 exists
  4. curl http://127.0.0.1:10550/api/v1/namespaces/default/configmaps?watch=ture

https://github.com/kubeedge/kubeedge/blob/ee357a0d5341241143e88d45df99fde865c987de/cloud/pkg/cloudhub/dispatcher/message_dispatcher.go#L268-L277

cloudhub would check objectsync ObjectResourceVersion with cm-1‘s ResourceVersion to decide send msg to edge or not.

neiba commented 1 year ago

cloudcore.yaml

apiVersion: cloudcore.config.kubeedge.io/v1alpha1
commonConfig:
  monitorServer:
    bindAddress: 127.0.0.1:9091
  tunnelPort: 10350
kind: CloudCore
kubeAPIConfig:
  burst: 5000
  contentType: application/vnd.kubernetes.protobuf
  kubeConfig: /root/.kube/config
  master: ""
  qps: 2500
modules:
  cloudHub:
    advertiseAddress:
    - 192.168.16.128
    dnsNames:
    - ""
    edgeCertSigningDuration: 365
    enable: true
    https:
      address: 0.0.0.0
      enable: true
      port: 10002
    keepaliveInterval: 30
    nodeLimit: 500
    quic:
      address: 0.0.0.0
      enable: false
      maxIncomingStreams: 10000
      port: 10001
    tlsCAFile: /tmp/etc/kubeedge/ca/cloudhub/rootCA.crt
    tlsCAKeyFile: /tmp/etc/kubeedge/ca/cloudhub/rootCA.key
    tlsCertFile: /tmp/etc/kubeedge/certs/cloudhub/server.crt
    tlsPrivateKeyFile: /tmp/etc/kubeedge/certs/cloudhub/server.key
    tokenRefreshDuration: 12
    unixsocket:
      address: unix:///tmp/var/lib/kubeedge/kubeedge.sock
      enable: true
    websocket:
      address: 0.0.0.0
      enable: true
      port: 10000
    writeTimeout: 30
  cloudStream:
    enable: true
    streamPort: 10003
    tlsStreamCAFile: /tmp/etc/kubeedge/ca/streamCA.crt
    tlsStreamCertFile: /tmp/etc/kubeedge/certs/stream.crt
    tlsStreamPrivateKeyFile: /tmp/etc/kubeedge/certs/stream.key
    tlsTunnelCAFile: /tmp/etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /tmp/etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /tmp/etc/kubeedge/certs/server.key
    tunnelPort: 10004
  deviceController:
    buffer:
      deviceEvent: 1
      deviceModelEvent: 1
      updateDeviceStatus: 1024
    enable: true
    load:
      updateDeviceStatusWorkers: 1
  dynamicController:
    enable: true
  edgeController:
    buffer:
      configMapEvent: 1
      createLease: 1524
      createNode: 1024
      deletePod: 1024
      patchNode: 1274
      patchPod: 1024
      podEvent: 1
      queryConfigMap: 1024
      queryLease: 1024
      queryNode: 1524
      queryPersistentVolume: 1024
      queryPersistentVolumeClaim: 1024
      querySecret: 1024
      queryVolumeAttachment: 1024
      ruleEndpointsEvent: 1
      rulesEvent: 1
      secretEvent: 1
      serviceAccountToken: 1024
      updateNode: 1024
      updateNodeStatus: 1024
      updatePodStatus: 1024
    enable: true
    load:
      ServiceAccountTokenWorkers: 100
      UpdateRuleStatusWorkers: 4
      createLeaseWorkers: 500
      createNodeWorkers: 100
      deletePodWorkers: 100
      patchNodeWorkers: 110
      patchPodWorkers: 100
      queryConfigMapWorkers: 100
      queryLeaseWorkers: 100
      queryNodeWorkers: 500
      queryPersistentVolumeClaimWorkers: 4
      queryPersistentVolumeWorkers: 4
      querySecretWorkers: 100
      queryVolumeAttachmentWorkers: 4
      updateNodeStatusWorkers: 1
      updateNodeWorkers: 4
      updatePodStatusWorkers: 1
    nodeUpdateFrequency: 10
  iptablesManager:
    enable: true
    mode: internal
  nodeUpgradeJobController:
    buffer:
      nodeUpgradeJobEvent: 1
      updateNodeUpgradeJobStatus: 1024
    enable: false
    load:
      nodeUpgradeJobWorkers: 1
  router:
    address: 0.0.0.0
    enable: true
    port: 9443
    restTimeout: 60
  syncController:
    enable: true
neiba commented 1 year ago

edgecore.yaml

apiVersion: edgecore.config.kubeedge.io/v1alpha2
database:
  aliasName: default
  dataSource: /tmp/var/lib/kubeedge/edgecore.db
  driverName: sqlite3
kind: EdgeCore
modules:
  dbTest:
    enable: false
  deviceTwin:
    enable: true
  edgeHub:
    enable: true
    heartbeat: 15
    httpServer: https://192.168.16.128:10002
    messageBurst: 60
    messageQPS: 30
    projectID: e632aba927ea4ac2b575ec1603d56f10
    quic:
      enable: false
      handshakeTimeout: 30
      readDeadline: 15
      server: 192.168.16.128:10001
      writeDeadline: 15
    rotateCertificates: true
    tlsCaFile: /tmp/etc/kubeedge/ca/rootCA.crt
    tlsCertFile: /tmp/etc/kubeedge/certs/server.crt
    tlsPrivateKeyFile: /tmp/etc/kubeedge/certs/server.key
    token: ""
    websocket:
      enable: true
      handshakeTimeout: 30
      readDeadline: 15
      server: 192.168.16.128:10000
      writeDeadline: 15
  edgeStream:
    enable: true
    handshakeTimeout: 30
    readDeadline: 15
    server: 127.0.0.1:10004
    tlsTunnelCAFile: /tmp/etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /tmp/etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /tmp/etc/kubeedge/certs/server.key
    writeDeadline: 15
  edged:
    cniBinDir: /opt/cni/bin
    cniCacheDir: /var/lib/cni/cache
    cniConfDir: /tmp/etc/cni/net.d
    containerRuntime: docker
    enable: true
    hostnameOverride: edge-node
    imagePullProgressDeadline: 1m0s
    masterServiceNamespace: default
    maxContainerCount: -1
    maxPerPodContainerCount: 1
    minimumGCAge: 0s
    networkPluginMTU: 1500
    nodeIP: 192.168.16.128
    nonMasqueradeCidr: 10.0.0.0/8
    podSandboxImage: kubeedge/pause:3.6
    registerNode: true
    registerNodeNamespace: default
    registerSchedulable: true
    remoteImageEndpoint: unix:///var/run/dockershim.sock
    remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
    rootDirectory: /var/lib/edged
    seccompProfileRoot: /var/lib/edged/seccomp
    tailoredKubeletConfig:
      address: 127.0.0.1
      cgroupDriver: cgroupfs
      cgroupsPerQOS: true
      clusterDomain: cluster.local
      configMapAndSecretChangeDetectionStrategy: Get
      containerLogMaxFiles: 5
      containerLogMaxSize: 10Mi
      contentType: application/json
      cpuCFSQuota: true
      cpuCFSQuotaPeriod: 100ms
      cpuManagerPolicy: none
      cpuManagerReconcilePeriod: 10s
      enableControllerAttachDetach: true
      enableDebugFlagsHandler: true
      enableDebuggingHandlers: true
      enableProfilingHandler: true
      enableSystemLogHandler: true
      enforceNodeAllocatable:
      - pods
      evictionHard:
        imagefs.available: 15%
        memory.available: 100Mi
        nodefs.available: 10%
        nodefs.inodesFree: 5%
      evictionPressureTransitionPeriod: 5m0s
      failSwapOn: false
      hairpinMode: promiscuous-bridge
      imageGCHighThresholdPercent: 80
      imageGCLowThresholdPercent: 40
      imageMinimumGCAge: 720h0m0s
      iptablesDropBit: 15
      iptablesMasqueradeBit: 14
      logging:
        flushFrequency: 5000000000
        format: text
        options:
          json:
            infoBufferSize: "0"
        verbosity: 0
      makeIPTablesUtilChains: true
      maxOpenFiles: 1000000
      maxPods: 110
      memoryManagerPolicy: None
      memorySwap: {}
      memoryThrottlingFactor: 0.8
      nodeLeaseDurationSeconds: 40
      nodeStatusMaxImages: 50
      nodeStatusReportFrequency: 5m0s
      nodeStatusUpdateFrequency: 10s
      oomScoreAdj: -999
      podPidsLimit: -1
      readOnlyPort: 10350
      registerNode: true
      registryBurst: 10
      registryPullQPS: 5
      resolvConf: /etc/resolv.conf
      runtimeRequestTimeout: 2m0s
      seccompDefault: false
      serializeImagePulls: true
      shutdownGracePeriod: 0s
      shutdownGracePeriodCriticalPods: 0s
      streamingConnectionIdleTimeout: 4h0m0s
      syncFrequency: 1m0s
      topologyManagerPolicy: none
      topologyManagerScope: container
      volumePluginDir: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
      volumeStatsAggPeriod: 1m0s
  eventBus:
    enable: true
    eventBusTLS:
      enable: false
      tlsMqttCAFile: /tmp/etc/kubeedge/ca/rootCA.crt
      tlsMqttCertFile: /tmp/etc/kubeedge/certs/server.crt
      tlsMqttPrivateKeyFile: /tmp/etc/kubeedge/certs/server.key
    mqttMode: 0
    mqttPassword: ""
    mqttPubClientID: ""
    mqttQOS: 0
    mqttRetain: false
    mqttServerExternal: tcp://127.0.0.1:1883
    mqttServerInternal: tcp://127.0.0.1:1884
    mqttSessionQueueSize: 100
    mqttSubClientID: ""
    mqttUsername: ""
  metaManager:
    contextSendGroup: hub
    contextSendModule: websocket
    enable: true
    metaServer:
      enable: true
      server: 127.0.0.1:10550
      tlsCaFile: /etc/kubeedge/ca/rootCA.crt
      tlsCertFile: /tmp/etc/kubeedge/certs/server.crt
      tlsPrivateKeyFile: /tmp/etc/kubeedge/certs/server.key
    remoteQueryTimeout: 60
  serviceBus:
    enable: true
    port: 9060
    server: 127.0.0.1
    timeout: 60
neiba commented 1 year ago

pod and configmap yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-1
data:
  test.txt: |
    test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: Exists
      containers:
      - name: nginx
        image: nginx:1.14.2
        volumeMounts:
        - name: vol-1
          mountPath: /vol-1
        ports:
        - containerPort: 80
          hostPort: 8080
      volumes:
      - name: vol-1
        configMap:
          name: cm-1
Shelley-BaoYue commented 1 year ago

Why path of metaManager.metaServer.tlsCaFile in egdecore.yaml is not save with others? I'm not sure if it caused the problem.

neiba commented 1 year ago

Why path of metaManager.metaServer.tlsCaFile in egdecore.yaml is not save with others? I'm not sure if it caused the problem.

metaserver is working, curl without watch is ok. I think this is cloudcore problem, cloudhub would check objectsync ObjectResourceVersion with cm-1‘s ResourceVersion to decide send msg to edge or not.

https://github.com/kubeedge/kubeedge/blob/ee357a0d5341241143e88d45df99fde865c987de/cloud/pkg/cloudhub/dispatcher/message_dispatcher.go#L268-L277

Shelley-BaoYue commented 1 year ago

cc @wackxu