cpu pinning issue - Githubissues

yangxiaoliang commented 1 year ago

What happened: Created 5 VMs with flavor 4c/4g/100g on a host, once all the 5 VMs started, then execute command taskset -pc ${vm-pid} in order to get the cpu pinning info, but I got an unexpected result as the vCPU of all the 5 VMs were pinned to the first 2 cores. The result shown in below picture:

What you expected to happen: The vCPU of VM can be pinned to different cores OR Don't pinning.

How to reproduce it (as minimally and precisely as possible):

Create 5 VMs with yaml like below:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    hulk.alpha.kubernetes.io/hulk-task-id: "44179910"
    hulk.alpha.kubernetes.io/k8s-task-id: "40150045"
    hulk.alpha.kubernetes.io/netStackType: ipv4
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
    trace.hulk.sankuai.com/m-spanid: 0.2.4.1
    trace.hulk.sankuai.com/m-traceid: "6904718154192519635"
  creationTimestamp: "2023-07-18T11:01:45Z"
  generation: 1
  labels:
    appkey: com.sankuai.shield.test.kata
    env: dev
    idc: hh
  name: vm-hh-shield-test-kata-dev05
  namespace: meituan-hulk
  resourceVersion: "1125720141"
  selfLink: /apis/kubevirt.io/v1/namespaces/meituan-hulk/virtualmachines/vm-hh-shield-test-kata-dev05
  uid: b7ba0015-6ad5-4df3-9326-bdbf255c85f0
spec:
  dataVolumeTemplates:
  - metadata:
      annotations:
        appkey: com.sankuai.shield.test.kata
      creationTimestamp: null
      name: sys-vm-hh-shield-test-kata-dev05
      namespace: meituan-hulk
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: csi-lvm-sc
      source:
        registry:
          secretRef: endpoint-secret
          url: docker://registry-hulk.sankuai.com/custom_dev/com.sankuai.hulknode.kubelet.union/centos-7-20230410
  - metadata:
      annotations:
        appkey: com.sankuai.shield.test.kata
      creationTimestamp: null
      name: swap-vm-hh-shield-test-kata-dev05
      namespace: meituan-hulk
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2Gi
        storageClassName: csi-lvm-sc
      source:
        blank: {}
  - metadata:
      annotations:
        appkey: com.sankuai.shield.test.kata
      creationTimestamp: null
      name: data-vm-hh-shield-test-kata-dev05
      namespace: meituan-hulk
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 40Gi
        storageClassName: csi-lvm-sc
      source:
        blank: {}
  runStrategy: RerunOnFailure
  template:
    metadata:
      annotations:
        hulk.alpha.kubernetes.io/hulk-task-id: "44179910"
        hulk.alpha.kubernetes.io/k8s-task-id: "40150045"
        hulk.alpha.kubernetes.io/netStackType: ipv4
        trace.hulk.sankuai.com/m-spanid: 0.2.4.1
        trace.hulk.sankuai.com/m-traceid: "6904718154192519635"
      creationTimestamp: null
      labels:
        appkey: com.sankuai.shield.test.kata
        env: dev
        idc: hh
      name: vm-hh-shield-test-kata-dev05
      namespace: meituan-hulk
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubevirt.io/schedulable
                operator: In
                values:
                - "true"
              - key: hostStatus
                operator: In
                values:
                - online
              - key: idc
                operator: In
                values:
                - hh
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: appkey
                  operator: In
                  values:
                  - com.sankuai.shield.test.kata
              topologyKey: kubernetes.io/hostname
            weight: 100
      domain:
        cpu:
          cores: 4
        devices:
          disks:
          - disk:
              bus: virtio
            name: sys-vm-hh-shield-test-kata-dev05
          - disk:
              bus: virtio
            name: swap-vm-hh-shield-test-kata-dev05
          - disk:
              bus: virtio
            name: data-vm-hh-shield-test-kata-dev05
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - bridge: {}
            macAddress: 00:42:8a:a8:d4:8c
            name: default
        machine:
          type: q35
        resources:
          limits:
            cpu: "4"
            memory: 4Gi
          requests:
            cpu: "1"
            memory: 4Gi
      networks:
      - name: default
        pod: {}
      volumes:
      - dataVolume:
          name: sys-vm-hh-shield-test-kata-dev05
        name: sys-vm-hh-shield-test-kata-dev05
      - dataVolume:
          name: swap-vm-hh-shield-test-kata-dev05
        name: swap-vm-hh-shield-test-kata-dev05
      - dataVolume:
          name: data-vm-hh-shield-test-kata-dev05
        name: data-vm-hh-shield-test-kata-dev05
      - cloudInitNoCloud:
          userDataBase64: I2Nsb3VkLWNvbmZpZwpob3N0bmFtZTogdm0taGgtc2hpZWxkLXRlc3Qta2F0YS1kZXYwNS5tdApkaXNrX3NldHVwOgogIC9kZXYvdmRiOgogICAgdGFibGVfdHlwZTogZ3B0CiAgICBsYXlvdXQ6IFRydWUKICAgIG92ZXJ3cml0ZTogVHJ1ZQoKICAvZGV2L3ZkYzoKICAgIHRhYmxlX3R5cGU6IGdwdAogICAgbGF5b3V0OiBUcnVlCgpmc19zZXR1cDoKICAtIGxhYmVsOiAvZGV2L3ZkYgogICAgZmlsZXN5c3RlbTogc3dhcAogICAgZGV2aWNlOiAvZGV2L3ZkYjEKCiAgLSBsYWJlbDogL2Rldi92ZGMKICAgIGZpbGVzeXN0ZW06IGV4dDQKICAgIGRldmljZTogL2Rldi92ZGMxCgptb3VudHM6CiAtIFsgIi9kZXYvdmRiMSIsICJub25lIiwgInN3YXAiLCAic3csbm9mYWlsIiwgIjAiLCAiMCIgXQogLSBbICIvZGV2L3ZkYzEiLCAiL29wdCIsICJleHQ0IiwgImRlZmF1bHRzLG5vZmFpbCIsICIwIiwgIjAiIF0KCmJvb3RjbWQ6CiAtIGVjaG8gSE9TVE5BTUU9dm0taGgtc2hpZWxkLXRlc3Qta2F0YS1kZXYwNS5tdCA+IC9ldGMvc3lzY29uZmlnL25ldHdvcmsKIC0gZWNobyBORVRXT1JLSU5HPXllcyA+PiAvZXRjL3N5c2NvbmZpZy9uZXR3b3JrCiAtIGVjaG8gIiIgPj4gL2V0Yy9ob3N0cwogLSBbIGNsb3VkLWluaXQtcGVyLCBpbnN0YW5jZSwgaW5pdCwgL2Jpbi9zZWQsIC1pLCAnMWkgMTI3LjAuMC4xIGxvY2FsaG9zdCcsIC9ldGMvaG9zdHMgXQogLSAvYmluL3NlZCAtaSAnJHsvXlxzKiQvZH0nIC9ldGMvaG9zdHMKIC0gL3NiaW4vZXRodG9vbCAtcyBldGgwIHNwZWVkIDEwMDAwCiAtIGVjaG8gbmFtZXNlcnZlciAxMC4xMC4xMC4zMCA+PiAvZXRjL3Jlc29sdi5jb25mCiAtIGVjaG8gbmFtZXNlcnZlciAxMC4xMC4xMC40MCA+PiAvZXRjL3Jlc29sdi5jb25mCg==
        name: cloudinitdisk
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:02:02Z"
    status: "True"
    type: Ready
  created: true
  printableStatus: Running
  ready: true
  volumeSnapshotStatuses:
  - enabled: false
    name: sys-vm-hh-shield-test-kata-dev05
    reason: 'No VolumeSnapshotClass: Volume snapshots are not configured for this
      StorageClass [csi-lvm-sc] [sys-vm-hh-shield-test-kata-dev05]'
  - enabled: false
    name: swap-vm-hh-shield-test-kata-dev05
    reason: 'No VolumeSnapshotClass: Volume snapshots are not configured for this
      StorageClass [csi-lvm-sc] [swap-vm-hh-shield-test-kata-dev05]'
  - enabled: false
    name: data-vm-hh-shield-test-kata-dev05
    reason: 'No VolumeSnapshotClass: Volume snapshots are not configured for this
      StorageClass [csi-lvm-sc] [data-vm-hh-shield-test-kata-dev05]'
  - enabled: false
    name: cloudinitdisk
    reason: Volume type has no StorageClass defined

Additional context:

VMI Yaml:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  annotations:
    hulk.alpha.kubernetes.io/hulk-task-id: "44179910"
    hulk.alpha.kubernetes.io/k8s-task-id: "40150045"
    hulk.alpha.kubernetes.io/netStackType: ipv4
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1alpha3
    trace.hulk.sankuai.com/m-spanid: 0.2.4.1
    trace.hulk.sankuai.com/m-traceid: "6904718154192519635"
  creationTimestamp: "2023-07-18T11:01:48Z"
  finalizers:
  - foregroundDeleteVirtualMachine
  generation: 13
  labels:
    appkey: com.sankuai.shield.test.kata
    env: dev
    idc: hh
    kubevirt.io/nodeName: hh-hulk-k8s-ep-test189.mt
  name: vm-hh-shield-test-kata-dev05
  namespace: meituan-hulk
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachine
    name: vm-hh-shield-test-kata-dev05
    uid: b7ba0015-6ad5-4df3-9326-bdbf255c85f0
  resourceVersion: "1125721425"
  selfLink: /apis/kubevirt.io/v1/namespaces/meituan-hulk/virtualmachineinstances/vm-hh-shield-test-kata-dev05
  uid: 0989b28e-cb57-4097-bd05-40174ab9c3f7
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubevirt.io/schedulable
            operator: In
            values:
            - "true"
          - key: hostStatus
            operator: In
            values:
            - online
          - key: idc
            operator: In
            values:
            - hh
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: appkey
              operator: In
              values:
              - com.sankuai.shield.test.kata
          topologyKey: kubernetes.io/hostname
        weight: 100
  domain:
    cpu:
      cores: 4
    devices:
      disks:
      - disk:
          bus: virtio
        name: sys-vm-hh-shield-test-kata-dev05
      - disk:
          bus: virtio
        name: swap-vm-hh-shield-test-kata-dev05
      - disk:
          bus: virtio
        name: data-vm-hh-shield-test-kata-dev05
      - disk:
          bus: virtio
        name: cloudinitdisk
      interfaces:
      - bridge: {}
        macAddress: 00:42:8a:a8:d4:8c
        name: default
    features:
      acpi:
        enabled: true
    firmware:
      uuid: 712646eb-ba57-51a4-9937-01995de3c8a1
    machine:
      type: q35
    resources:
      limits:
        cpu: "4"
        memory: 4Gi
      requests:
        cpu: "1"
        memory: 4Gi
  networks:
  - name: default
    pod: {}
  volumes:
  - dataVolume:
      name: sys-vm-hh-shield-test-kata-dev05
    name: sys-vm-hh-shield-test-kata-dev05
  - dataVolume:
      name: swap-vm-hh-shield-test-kata-dev05
    name: swap-vm-hh-shield-test-kata-dev05
  - dataVolume:
      name: data-vm-hh-shield-test-kata-dev05
    name: data-vm-hh-shield-test-kata-dev05
  - cloudInitNoCloud:
      userDataBase64: I2Nsb3VkLWNvbmZpZwpob3N0bmFtZTogdm0taGgtc2hpZWxkLXRlc3Qta2F0YS1kZXYwNS5tdApkaXNrX3NldHVwOgogIC9kZXYvdmRiOgogICAgdGFibGVfdHlwZTogZ3B0CiAgICBsYXlvdXQ6IFRydWUKICAgIG92ZXJ3cml0ZTogVHJ1ZQoKICAvZGV2L3ZkYzoKICAgIHRhYmxlX3R5cGU6IGdwdAogICAgbGF5b3V0OiBUcnVlCgpmc19zZXR1cDoKICAtIGxhYmVsOiAvZGV2L3ZkYgogICAgZmlsZXN5c3RlbTogc3dhcAogICAgZGV2aWNlOiAvZGV2L3ZkYjEKCiAgLSBsYWJlbDogL2Rldi92ZGMKICAgIGZpbGVzeXN0ZW06IGV4dDQKICAgIGRldmljZTogL2Rldi92ZGMxCgptb3VudHM6CiAtIFsgIi9kZXYvdmRiMSIsICJub25lIiwgInN3YXAiLCAic3csbm9mYWlsIiwgIjAiLCAiMCIgXQogLSBbICIvZGV2L3ZkYzEiLCAiL29wdCIsICJleHQ0IiwgImRlZmF1bHRzLG5vZmFpbCIsICIwIiwgIjAiIF0KCmJvb3RjbWQ6CiAtIGVjaG8gSE9TVE5BTUU9dm0taGgtc2hpZWxkLXRlc3Qta2F0YS1kZXYwNS5tdCA+IC9ldGMvc3lzY29uZmlnL25ldHdvcmsKIC0gZWNobyBORVRXT1JLSU5HPXllcyA+PiAvZXRjL3N5c2NvbmZpZy9uZXR3b3JrCiAtIGVjaG8gIiIgPj4gL2V0Yy9ob3N0cwogLSBbIGNsb3VkLWluaXQtcGVyLCBpbnN0YW5jZSwgaW5pdCwgL2Jpbi9zZWQsIC1pLCAnMWkgMTI3LjAuMC4xIGxvY2FsaG9zdCcsIC9ldGMvaG9zdHMgXQogLSAvYmluL3NlZCAtaSAnJHsvXlxzKiQvZH0nIC9ldGMvaG9zdHMKIC0gL3NiaW4vZXRodG9vbCAtcyBldGgwIHNwZWVkIDEwMDAwCiAtIGVjaG8gbmFtZXNlcnZlciAxMC4xMC4xMC4zMCA+PiAvZXRjL3Jlc29sdi5jb25mCiAtIGVjaG8gbmFtZXNlcnZlciAxMC4xMC4xMC40MCA+PiAvZXRjL3Jlc29sdi5jb25mCg==
    name: cloudinitdisk
status:
  activePods:
    e1a0d793-0f21-4aba-98e8-57556cf9bb7d: hh-hulk-k8s-ep-test189.mt
  conditions:
  - lastProbeTime: null
    lastTransitionTime: null
    message: 'cannot migrate VMI: PVC sys-vm-hh-shield-test-kata-dev05 is not shared,
      live migration requires that all PVCs must be shared (using ReadWriteMany access
      mode)'
    reason: DisksNotLiveMigratable
    status: "False"
    type: LiveMigratable
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:02:02Z"
    status: "True"
    type: Ready
  - lastProbeTime: "2023-07-18T11:03:35Z"
    lastTransitionTime: null
    status: "True"
    type: AgentConnected
  guestOSInfo:
    id: centos
    kernelRelease: 4.18.0-147.mt20200626.413.el8_1.x86_64
    kernelVersion: '#1 SMP Fri Jun 26 10:27:19 CST 2020'
    name: CentOS Linux
    prettyName: CentOS Linux 7 (Core)
    version: "7"
    versionId: "7"
  interfaces:
  - interfaceName: eth0
    ipAddress: 10.171.175.182
    ipAddresses:
    - 10.171.175.182
    mac: 00:42:8a:a8:d4:8c
    name: default
  launcherContainerImageVersion: registry-hulk.sankuai.com/custom_dev/com.sankuai.hulknode.kubelet.union/virt-launcher:v0.43.0
  migrationMethod: BlockMigration
  nodeName: hh-hulk-k8s-ep-test189.mt
  phase: Running
  phaseTransitionTimestamps:
  - phase: Scheduling
    phaseTransitionTimestamp: "2023-07-18T11:02:57Z"
  - phase: Scheduled
    phaseTransitionTimestamp: "2023-07-18T11:02:57Z"
  - phase: Running
    phaseTransitionTimestamp: "2023-07-18T11:03:03Z"
  qosClass: Burstable
  volumeStatus:
  - name: cloudinitdisk
    target: vdd
  - name: data-vm-hh-shield-test-kata-dev05
    target: vdc
  - name: swap-vm-hh-shield-test-kata-dev05
    target: vdb
  - name: sys-vm-hh-shield-test-kata-dev05
    target: vda

virt-launcher Pod Yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    hulk.alpha.kubernetes.io/hulk-task-id: "44179910"
    hulk.alpha.kubernetes.io/k8s-task-id: "40150045"
    hulk.alpha.kubernetes.io/netStackType: ipv4
    kubevirt.io/domain: vm-hh-shield-test-kata-dev05
    trace.hulk.sankuai.com/m-spanid: 0.2.4.1
    trace.hulk.sankuai.com/m-traceid: "6904718154192519635"
  creationTimestamp: "2023-07-18T11:01:49Z"
  labels:
    appkey: com.sankuai.shield.test.kata
    env: dev
    idc: hh
    kubevirt.io: virt-launcher
    kubevirt.io/created-by: 0989b28e-cb57-4097-bd05-40174ab9c3f7
  name: virt-launcher-vm-hh-shield-test-kata-dev05-mt
  namespace: meituan-hulk
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: VirtualMachineInstance
    name: vm-hh-shield-test-kata-dev05
    uid: 0989b28e-cb57-4097-bd05-40174ab9c3f7
  resourceVersion: "1125720000"
  selfLink: /api/v1/namespaces/meituan-hulk/pods/virt-launcher-vm-hh-shield-test-kata-dev05-mt
  uid: e1a0d793-0f21-4aba-98e8-57556cf9bb7d
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubevirt.io/schedulable
            operator: In
            values:
            - "true"
          - key: hostStatus
            operator: In
            values:
            - online
          - key: idc
            operator: In
            values:
            - hh
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: appkey
              operator: In
              values:
              - com.sankuai.shield.test.kata
          topologyKey: kubernetes.io/hostname
        weight: 100
  automountServiceAccountToken: false
  containers:
  - command:
    - /usr/bin/virt-launcher
    - --qemu-timeout
    - 15m
    - --name
    - vm-hh-shield-test-kata-dev05
    - --uid
    - 0989b28e-cb57-4097-bd05-40174ab9c3f7
    - --namespace
    - meituan-hulk
    - --kubevirt-share-dir
    - /var/run/kubevirt
    - --ephemeral-disk-dir
    - /var/run/kubevirt-ephemeral-disks
    - --container-disk-dir
    - /var/run/kubevirt/container-disks
    - --grace-period-seconds
    - "45"
    - --hook-sidecars
    - "0"
    - --less-pvc-space-toleration
    - "10"
    - --minimum-pvc-reserve-bytes
    - "131072"
    - --ovmf-path
    - /usr/share/OVMF
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    image: registry-hulk.sankuai.com/custom_dev/com.sankuai.hulknode.kubelet.union/virt-launcher:v0.43.0
    imagePullPolicy: IfNotPresent
    name: compute
    resources:
      limits:
        cpu: "4"
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        memory: "4506779649"
      requests:
        cpu: "1"
        devices.kubevirt.io/kvm: "1"
        devices.kubevirt.io/tun: "1"
        devices.kubevirt.io/vhost-net: "1"
        ephemeral-storage: 50M
        memory: "4506779649"
    securityContext:
      capabilities:
        add:
        - SYS_NICE
        drop:
        - NET_RAW
      privileged: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/kubevirt-ephemeral-disks
      name: ephemeral-disks
    - mountPath: /var/run/kubevirt/container-disks
      mountPropagation: HostToContainer
      name: container-disks
    - mountPath: /var/run/kubevirt/hotplug-disks
      mountPropagation: HostToContainer
      name: hotplug-disks
    - mountPath: /var/run/libvirt
      name: libvirt-runtime
    - mountPath: /var/run/kubevirt/sockets
      name: sockets
    - mountPath: /var/run/kubevirt-private/vmi-disks/sys-vm-hh-shield-test-kata-dev05
      name: sys-vm-hh-shield-test-kata-dev05
    - mountPath: /var/run/kubevirt-private/vmi-disks/swap-vm-hh-shield-test-kata-dev05
      name: swap-vm-hh-shield-test-kata-dev05
    - mountPath: /var/run/kubevirt-private/vmi-disks/data-vm-hh-shield-test-kata-dev05
      name: data-vm-hh-shield-test-kata-dev05
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  hostname: vm-hh-shield-test-kata-dev05
  nodeName: hh-hulk-k8s-ep-test189.mt
  nodeSelector:
    kubevirt.io/schedulable: "true"
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext:
    runAsUser: 0
    seLinuxOptions:
      type: virt_launcher.process
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 60
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: sockets
  - name: sys-vm-hh-shield-test-kata-dev05
    persistentVolumeClaim:
      claimName: sys-vm-hh-shield-test-kata-dev05
  - name: swap-vm-hh-shield-test-kata-dev05
    persistentVolumeClaim:
      claimName: swap-vm-hh-shield-test-kata-dev05
  - name: data-vm-hh-shield-test-kata-dev05
    persistentVolumeClaim:
      claimName: data-vm-hh-shield-test-kata-dev05
  - emptyDir: {}
    name: virt-bin-share-dir
  - emptyDir: {}
    name: libvirt-runtime
  - emptyDir: {}
    name: ephemeral-disks
  - emptyDir: {}
    name: container-disks
  - emptyDir: {}
    name: hotplug-disks
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:01:50Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:02:02Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:02:02Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-18T11:01:50Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://2ba531292ea7d039d9938891dcaa3f24b795f1bc4ad5615f4a1bffb02ea78c61
    image: docker.io/custom_dev/com.sankuai.hulknode.kubelet.union/virt-launcher:v0.43.0
    imageID: docker-pullable://docker.io/custom_dev/com.sankuai.hulknode.kubelet.union/virt-launcher@sha256:1eff676608a314e44ee1092355e2f260901395a22bc64fd9f08fa4d4843d2044
    lastState: {}
    name: compute
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-07-18T11:02:02Z"
  hostIP: 10.171.81.46
  phase: Running
  podIP: 10.171.175.182
  podIPs:
  - ip: 10.171.175.182
  qosClass: Burstable
  startTime: "2023-07-18T11:01:50Z"

VM XML

  <name>meituan-hulk_vm-hh-shield-test-kata-dev05</name>
  <uuid>712646eb-ba57-51a4-9937-01995de3c8a1</uuid>
  <metadata>
    <kubevirt xmlns="http://kubevirt.io">
      <uid>0989b28e-cb57-4097-bd05-40174ab9c3f7</uid>
      <graceperiod>
        <deletionGracePeriodSeconds>30</deletionGracePeriodSeconds>
      </graceperiod>
    </kubevirt>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <iothreads>1</iothreads>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>KubeVirt</entry>
      <entry name='product'>None</entry>
      <entry name='uuid'>712646eb-ba57-51a4-9937-01995de3c8a1</entry>
      <entry name='family'>KubeVirt</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel8.4.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <topology sockets='1' dies='1' cores='4' threads='1'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='require' name='tsx-ctrl'/>
    <feature policy='disable' name='mpx'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-private/vmi-disks/sys-vm-hh-shield-test-kata-dev05/disk.img' index='4'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='ua-sys-vm-hh-shield-test-kata-dev05'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-private/vmi-disks/swap-vm-hh-shield-test-kata-dev05/disk.img' index='3'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='ua-swap-vm-hh-shield-test-kata-dev05'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-private/vmi-disks/data-vm-hh-shield-test-kata-dev05/disk.img' index='2'/>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
      <alias name='ua-data-vm-hh-shield-test-kata-dev05'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap'/>
      <source file='/var/run/kubevirt-ephemeral-disks/cloud-init-data/meituan-hulk/vm-hh-shield-test-kata-dev05/noCloud.iso' index='1'/>
      <backingStore/>
      <target dev='vdd' bus='virtio'/>
      <alias name='ua-cloudinitdisk'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-non-transitional'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0' model='virtio-non-transitional'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='ethernet'>
      <mac address='00:42:8a:a8:d4:8c'/>
      <target dev='tap0' managed='no'/>
      <model type='virtio-non-transitional'/>
      <mtu size='1500'/>
      <alias name='ua-default'/>
      <rom enabled='no'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/0989b28e-cb57-4097-bd05-40174ab9c3f7/virt-serial0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='unix'>
      <source mode='bind' path='/var/run/kubevirt-private/0989b28e-cb57-4097-bd05-40174ab9c3f7/virt-serial0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-meituan-hulk_vm-hh-s/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' socket='/var/run/kubevirt-private/0989b28e-cb57-4097-bd05-40174ab9c3f7/virt-vnc'>
      <listen type='socket' socket='/var/run/kubevirt-private/0989b28e-cb57-4097-bd05-40174ab9c3f7/virt-vnc'/>
    </graphics>
    <video>
      <model type='vga' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio-non-transitional'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

the output of lscpu from host:

Environment:

KubeVirt version (use virtctl version): v0.43.0
Kubernetes version (use kubectl version): 1.16
VM or VMI specifications:
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): centos7
Kernel (e.g. uname -a): Linux hh-hulk-k8s-ep-test189.mt 4.18.0-147.mt20200626.413.el8_1.x86_64 #1 SMP Fri Jun 26 10:27:19 CST 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools: N/A
Others: N/A

simonyangcj commented 1 year ago

Looks like you didn't tell vm to use dedicated cpu in the first place, you might want to add dedicatedCpuPlacement: true in spec.domain.cpu, then ensure your node affinity will correct filter out node that use cpu manager with static policy or your vm will be result in pending. Please see https://kubevirt.io/user-guide/virtual_machines/dedicated_cpu_resources/ for detail

yangxiaoliang commented 1 year ago

Looks like you didn't tell vm to use dedicated cpu in the first place, you might want to add dedicatedCpuPlacement: true in spec.domain.cpu, then ensure your node affinity will correct filter out node that use cpu manager with static policy or your vm will be result in pending. Please see https://kubevirt.io/user-guide/virtual_machines/dedicated_cpu_resources/ for detail

Thanks for your reply @simonyangcj , actually we don't want the cpu pinning happens ...

simonyangcj commented 1 year ago

my apologizes, this seems like might relate to kubernetes cpu manager. Does this host enable cpu manager. Please run cat /var/lib/kubelet/cpu_manager_state to checkout whether cpu manager is enabled

yangxiaoliang commented 1 year ago

my apologizes, this seems like might relate to kubernetes cpu manager. Does this host enable cpu manager. Please run cat /var/lib/kubelet/cpu_manager_state to checkout whether cpu manager is enabled

Paste the result as below:

this means the cpu manager is not eanbled ?

and from the cgroup settings, the virt-launcher pod not been pinned to any cores:

following is the kubelet start command:

simonyangcj commented 1 year ago

You r right, CPU manager is not enabled. But your kubelet config looks a bit odd, i checked kubelet 1.16 doesn't seem like there are a config --cpuset-cpus, I assume it might has something to do with it. Another possibility is OS level CPU isolation you might want give a check.

mlavi commented 1 year ago

We discussed this on the KubeVirt community meeting; it would be helpful if you upgrade to supported versions of K8s and KubeVirt to gain more community support. Please review https://github.com/kubevirt/kubevirt/blob/9cf6dbb3e33c9ec9e9af8b18eb115c7fb9f7c311/docs/release.md?plain=1#L27C19-L27C19

kubevirt-bot commented 10 months ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

kubevirt-bot commented 9 months ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

kubevirt-bot commented 8 months ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

/close

kubevirt-bot commented 8 months ago

@kubevirt-bot: Closing this issue.

In response to [this](https://github.com/kubevirt/kubevirt/issues/10132#issuecomment-1858845632): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubevirt / kubevirt

cpu pinning issue #10132