Provide possibility to configure uneven split of hugepages between NUMA nodes

ghost commented 4 years ago

Provide possibility to configure uneven split of hugepages between NUMA nodes.

At the moment, one can do either of the following 2:

a) Evenly split all hugepages between both NUMA nodes:

[root@openshift-jumpserver-0 config]# cat pp.yaml 
apiVersion: performance.openshift.io/v1alpha1
kind: PerformanceProfile
metadata:
  name: worker-cnf
spec:
  cpu:
    isolated: 0-8,20-28
    reserved: 9-19,29-39
  hugepages:
    defaultHugepagesSize: "1G"
    pages:
    - size: "1G"
      count: 32
      # node: 0
  realTimeKernel:
    enabled: false
  nodeSelector:
    node-role.kubernetes.io/worker-cnf: ""

b) Have all hugepages on a single NUMA node:

[root@openshift-jumpserver-0 config]# cat pp2.yaml 
apiVersion: performance.openshift.io/v1alpha1
kind: PerformanceProfile
metadata:
  name: worker-cnf2
spec:
  cpu:
    reserved: 0-8,20-28
    isolated: 9-19,29-39
  hugepages:
    defaultHugepagesSize: "1G"
    pages:
    - size: "1G"
      count: 32
      node: 1
  realTimeKernel:
    enabled: true
  nodeSelector:
    node-role.kubernetes.io/worker-cnf2: ""

c) The following configuration does not work, although I find this to be an intuitive setting:

[root@openshift-jumpserver-0 config]# cat pp2.yaml 
apiVersion: performance.openshift.io/v1alpha1
kind: PerformanceProfile
metadata:
  name: worker-cnf2
spec:
  cpu:
    reserved: 0-8,20-28
    isolated: 9-19,29-39
  hugepages:
    defaultHugepagesSize: "1G"
    pages:
    - size: "1G"
      count: 32
      node: 1
    - size: "1G"
      count: 16
      node: 0
  realTimeKernel:
    enabled: true
  nodeSelector:
    node-role.kubernetes.io/worker-cnf2: ""

Thanks,

Andreas

andreaskaris commented 4 years ago

a) works:

[root@openshift-worker-0 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-7edd22c1588471983b87af0873313ee8e17e775729360c9720c6558ce3a1f3b5/vmlinuz-4.18.0-147.8.1.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ostree=/ostree/boot.0/rhcos/7edd22c1588471983b87af0873313ee8e17e775729360c9720c6558ce3a1f3b5/0 intel_iommu=on iommu=pt nohz=on nosoftlockup skew_tick=1 intel_pstate=disable intel_iommu=on iommu=pt default_hugepagesz=1G rcu_nocbs=0-8,20-28 tuned.non_isolcpus=000000ff,e00ffe00 hugepagesz=1G hugepages=32 [root@openshift-worker-0 ~]# find /sys -name 'meminfo' | xargs grep -i huge /sys/devices/system/node/node0/meminfo:Node 0 AnonHugePages: 61440 kB /sys/devices/system/node/node0/meminfo:Node 0 ShmemHugePages: 0 kB /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total: 16 /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free: 16 /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp: 0 /sys/devices/system/node/node1/meminfo:Node 1 AnonHugePages: 266240 kB /sys/devices/system/node/node1/meminfo:Node 1 ShmemHugePages: 0 kB /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total: 16 /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free: 16 /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp: 0

b) works:

[root@openshift-worker-1 ~]# find /sys -name 'meminfo' | xargs grep -i huge /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total: 0 /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free: 0 /sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp: 0 /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total: 32 /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free: 32 /sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp: 0 [root@openshift-worker-1 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-0af4c1650341e299b269001f93f962e11abde2011bf0050c8925aec0b3f236ce/vmlinuz-4.18.0-147.8.1.rt24.101.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ostree=/ostree/boot.1/rhcos/0af4c1650341e299b269001f93f962e11abde2011bf0050c8925aec0b3f236ce/0 intel_iommu=on iommu=pt nohz=on nosoftlockup skew_tick=1 intel_pstate=disable intel_iommu=on iommu=pt rcu_nocbs=9-19,29-39 tuned.non_isolcpus=1ff001ff default_hugepagesz=1G

ghost commented 4 years ago

c) no idea how that math came to be, but:

[root@openshift-worker-1 ~]# find /sys -name '*meminfo*' | xargs grep -i huge
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total:    12
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free:     12
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp:      0
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total:    25
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free:     25
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp:      0

apiVersion: performance.openshift.io/v1alpha1
kind: PerformanceProfile
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"performance.openshift.io/v1alpha1","kind":"PerformanceProfile","metadata":{"annotations":{},"name":"worker-cnf2"},"spec":{"cpu":{"isolated":"9-19,29-39","reserved":"0-8,20-28"},"hugepages":{"defaultHugepagesSize":"1G","pages":[{"count":32,"node":1,"size":"1G"},{"count":16,"node":0,"size":"1G"}]},"nodeSelector":{"node-role.kubernetes.io/worker-cnf2":""},"realTimeKernel":{"enabled":true}}}
  creationTimestamp: "2020-06-23T21:10:30Z"
  finalizers:
  - foreground-deletion
  generation: 2
  name: worker-cnf2
  resourceVersion: "585879"
  selfLink: /apis/performance.openshift.io/v1alpha1/performanceprofiles/worker-cnf2
  uid: b15acd9f-cce4-49f5-81a6-ce128e23154e
spec:
  cpu:
    isolated: 9-19,29-39
    reserved: 0-8,20-28
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 32
      node: 1
      size: 1G
    - count: 16
      node: 0
      size: 1G
  nodeSelector:
    node-role.kubernetes.io/worker-cnf2: ""
  realTimeKernel:
    enabled: true
status:
  conditions:
  - lastHeartbeatTime: "2020-06-23T21:10:27Z"
    lastTransitionTime: "2020-06-23T21:10:27Z"
    status: "True"
    type: Available
  - lastHeartbeatTime: "2020-06-23T21:10:27Z"
    lastTransitionTime: "2020-06-23T21:10:27Z"
    status: "True"
    type: Upgradeable
  - lastHeartbeatTime: "2020-06-23T21:10:27Z"
    lastTransitionTime: "2020-06-23T21:10:27Z"
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2020-06-23T21:10:27Z"
    lastTransitionTime: "2020-06-23T21:10:27Z"
    status: "False"
    type: Degraded

[root@openshift-jumpserver-0 config]# oc get nodes
NAME                 STATUS   ROLES                AGE   VERSION
openshift-master-0   Ready    master               30h   v1.17.1
openshift-master-1   Ready    master               30h   v1.17.1
openshift-master-2   Ready    master               30h   v1.17.1
openshift-worker-0   Ready    worker,worker-cnf    30h   v1.17.1
openshift-worker-1   Ready    worker,worker-cnf2   30h   v1.17.1

alosadagrande commented 4 years ago

Just adding a different configuration with same unexpected results:

apiVersion: performance.openshift.io/v1alpha1
kind: PerformanceProfile
metadata:
  name: performance
spec:
  additionalKernelArgs:
  - nmi_watchdog=0
  - audit=0
  - mce=off
  - processor.max_cstate=1
  - idle=poll
  - intel_idle.max_cstate=0
  - nosmt
  - tsc=reliable
  cpu:
    isolated: 5-60
    reserved: 0-4
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 8 
      size: 1G
      node: 0
    - count: 8
      size: 1G
      node: 1
  nodeSelector:
    node-role.kubernetes.io/worker-cnf: ""
  numa:
    topologyPolicy: best-effort
  realTimeKernel:
    enabled: true

Result:

sh-4.4# find /sys -name '*meminfo*' | xargs grep -i huge
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total:     3
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free:      3
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp:      0
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total:     5
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free:      5
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp:      0

sh-4.4# cat /proc/meminfo  | grep -i huge
HugePages_Total:       8
HugePages_Free:        8
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:         8388608 kB

cynepco3hahue commented 4 years ago

@alosadagrande Can you show the MC generated by the performance profile? And also the amount of free memory that you have on the host.

alosadagrande commented 4 years ago

Sure @cynepco3hahue:

I can see both hugepage units for both nodes and looks correctly. However they are not applied successfully.

Worker has enough memory.

$ oc get mc -o yaml performance-performance
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  creationTimestamp: "2020-09-23T10:45:44Z"
  generation: 2
  labels:
    machineconfiguration.openshift.io/role: worker-cnf
  name: performance-performance
  ownerReferences:
  - apiVersion: performance.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PerformanceProfile
    name: performance
    uid: 25dfbb09-edf2-4ff5-88a7-654549a56631
  resourceVersion: "5546548"
  selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigs/performance-performance
  uid: 6ce81bb4-6102-4563-89ad-0ac4ba35358d
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1ldW8gcGlwZWZhaWwKClNZU1RFTV9DT05GSUdfRklMRT0iL2V0Yy9zeXN0ZW1kL3N5c3RlbS5jb25mIgpTWVNURU1fQ09ORklHX0NVU1RPTV9GSUxFPSIvZXRjL3N5c3RlbWQvc3lzdGVtLmNvbmYuZC9zZXRBZmZpbml0eS5jb25mIgoKaWYgWyAtZiAvZXRjL3N5c2NvbmZpZy9pcnFiYWxhbmNlIF0gJiYgWyAtZiAke1NZU1RFTV9DT05GSUdfQ1VTVE9NX0ZJTEV9IF0gICYmIHJwbS1vc3RyZWUgc3RhdHVzIC1iIHwgZ3JlcCAtcSAtZSAiJHtTWVNURU1fQ09ORklHX0ZJTEV9ICR7U1lTVEVNX0NPTkZJR19DVVNUT01fRklMRX0iICYmIGVncmVwIC13cSAiXklSUUJBTEFOQ0VfQkFOTkVEX0NQVVM9JHtSRVNFUlZFRF9DUFVfTUFTS19JTlZFUlR9IiAvZXRjL3N5c2NvbmZpZy9pcnFiYWxhbmNlOyB0aGVuCiAgICBlY2hvICJQcmUgYm9vdCB0dW5pbmcgY29uZmlndXJhdGlvbiBhbHJlYWR5IGFwcGxpZWQiCmVsc2UKICAgICNTZXQgSVJRIGJhbGFuY2UgYmFubmVkIGNwdXMKICAgIGlmIFsgISAtZiAvZXRjL3N5c2NvbmZpZy9pcnFiYWxhbmNlIF07IHRoZW4KICAgICAgICB0b3VjaCAvZXRjL3N5c2NvbmZpZy9pcnFiYWxhbmNlCiAgICBmaQoKICAgIGlmIGdyZXAgLWxzICJJUlFCQUxBTkNFX0JBTk5FRF9DUFVTPSIgL2V0Yy9zeXNjb25maWcvaXJxYmFsYW5jZTsgdGhlbgogICAgICAgIHNlZCAtaSAicy9eLipJUlFCQUxBTkNFX0JBTk5FRF9DUFVTPS4qJC9JUlFCQUxBTkNFX0JBTk5FRF9DUFVTPSR7UkVTRVJWRURfQ1BVX01BU0tfSU5WRVJUfS8iIC9ldGMvc3lzY29uZmlnL2lycWJhbGFuY2UKICAgIGVsc2UKICAgICAgICBlY2hvICJJUlFCQUxBTkNFX0JBTk5FRF9DUFVTPSR7UkVTRVJWRURfQ1BVX01BU0tfSU5WRVJUfSIgPj4vZXRjL3N5c2NvbmZpZy9pcnFiYWxhbmNlCiAgICBmaQoKICAgIHJwbS1vc3RyZWUgaW5pdHJhbWZzIC0tZW5hYmxlIC0tYXJnPS1JIC0tYXJnPSIke1NZU1RFTV9DT05GSUdfRklMRX0gJHtTWVNURU1fQ09ORklHX0NVU1RPTV9GSUxFfSIgCgogICAgdG91Y2ggL3Zhci9yZWJvb3QKZmkK
          verification: {}
        filesystem: root
        mode: 448
        path: /usr/local/bin/pre-boot-tuning.sh
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1ldW8gcGlwZWZhaWwKCm5vZGVzX3BhdGg9Ii9zeXMvZGV2aWNlcy9zeXN0ZW0vbm9kZSIKaHVnZXBhZ2VzX2ZpbGU9IiR7bm9kZXNfcGF0aH0vbm9kZSR7TlVNQV9OT0RFfS9odWdlcGFnZXMvaHVnZXBhZ2VzLSR7SFVHRVBBR0VTX1NJWkV9a0IvbnJfaHVnZXBhZ2VzIgoKaWYgWyAhIC1mICAke2h1Z2VwYWdlc19maWxlfSBdOyB0aGVuCiAgICBlY2hvICJFUlJPUjogJHtodWdlcGFnZXNfZmlsZX0gZG9lcyBub3QgZXhpc3QiCiAgICBleGl0IDEKZmkKCmVjaG8gJHtIVUdFUEFHRVNfQ09VTlR9ID4gJHtodWdlcGFnZXNfZmlsZX0KCmlmIFsgJChjYXQgJHtodWdlcGFnZXNfZmlsZX0pIC1uZSAke0hVR0VQQUdFU19DT1VOVH0gXTsgdGhlbgogICAgZWNobyAiRVJST1I6ICR7aHVnZXBhZ2VzX2ZpbGV9IGRvZXMgbm90IGhhdmUgdGhlIGV4cGVjdGVkIG51bWJlciBvZiBodWdlcGFnZXMgJHtIVUdFUEFHRVNfQ09VTlR9IgogICAgZXhpdCAxCmZpCg==
          verification: {}
        filesystem: root
        mode: 448
        path: /usr/local/bin/hugepages-allocation.sh
      - contents:
          source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1ldW8gcGlwZWZhaWwKCmlmIFtbIC1mIC92YXIvcmVib290IF1dOyB0aGVuIAogICAgcm0gLWYgL3Zhci9yZWJvb3QKICAgIGVjaG8gIkZpbGUgL3Zhci9yZWJvb3QgZXhpc3RzLCBpbml0aWF0ZSByZWJvb3QiCiAgICBzeXN0ZW1jdGwgcmVib290CmZpCg==
          verification: {}
        filesystem: root
        mode: 448
        path: /usr/local/bin/reboot.sh
      - contents:
          source: data:text/plain;charset=utf-8;base64,W01hbmFnZXJdCkNQVUFmZmluaXR5PTAtNA==
          verification: {}
        filesystem: root
        mode: 448
        path: /etc/systemd/system.conf.d/setAffinity.conf
    systemd:
      units:
      - contents: |
          [Unit]
          Description=Preboot tuning patch
          Before=kubelet.service
          Before=reboot.service

          [Service]
          Environment=RESERVED_CPUS=0-4
          Environment=RESERVED_CPU_MASK_INVERT=ffffffff,ffffffe0
          Type=oneshot
          RemainAfterExit=true
          ExecStart=/usr/local/bin/pre-boot-tuning.sh

          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: pre-boot-tuning.service
      - contents: |
          [Unit]
          Description=Reboot initiated by pre-boot-tuning
          Wants=network-online.target
          After=network-online.target
          Before=kubelet.service

          [Service]
          Type=oneshot
          RemainAfterExit=true
          ExecStart=/usr/local/bin/reboot.sh

          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: reboot.service
      - contents: |
          [Unit]
          Description=Hugepages-1048576kB allocation on the node 0
          Before=kubelet.service

          [Service]
          Environment=HUGEPAGES_COUNT=8
          Environment=HUGEPAGES_SIZE=1048576
          Environment=NUMA_NODE=0
          Type=oneshot
          RemainAfterExit=true
          ExecStart=/usr/local/bin/hugepages-allocation.sh

          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: hugepages-allocation-1048576kB-NUMA0.service
      - contents: |
          [Unit]
          Description=Hugepages-1048576kB allocation on the node 1
          Before=kubelet.service

          [Service]
          Environment=HUGEPAGES_COUNT=8
          Environment=HUGEPAGES_SIZE=1048576
          Environment=NUMA_NODE=1
          Type=oneshot
          RemainAfterExit=true
          ExecStart=/usr/local/bin/hugepages-allocation.sh

          [Install]
          WantedBy=multi-user.target
        enabled: true
        name: hugepages-allocation-1048576kB-NUMA1.service
  fips: false
  kernelArguments:
  - nohz=on
  - nosoftlockup
  - skew_tick=1
  - intel_pstate=disable
  - intel_iommu=on
  - iommu=pt
  - rcu_nocbs=5-60
  - tuned.non_isolcpus=0000001f
  - default_hugepagesz=1G
  - nmi_watchdog=0
  - audit=0
  - mce=off
  - processor.max_cstate=1
  - idle=poll
  - intel_idle.max_cstate=0
  - nosmt
  - tsc=reliable
  kernelType: realtime
  osImageURL: ""

sh-4.4# free -m
              total        used        free      shared  buff/cache   available
Mem:         385513       19835      360293          23        5384      365256
Swap:             0           0           0

cynepco3hahue commented 4 years ago

Yep, you totally have enough memory, can you please paste dmesg output?

alosadagrande commented 4 years ago

sh-4.4# dmesg
[    0.000000] microcode: microcode updated early to revision 0x2006906, date = 2020-04-24
[    0.000000] Linux version 4.18.0-193.14.3.rt13.67.el8_2.x86_64 (mockbuild@x86-vm-08.build.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP PREEMPT RT Mon Jul 20 16:41:14 UTC 2020
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/vmlinuz-4.18.0-193.14.3.rt13.67.el8_2.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.0/rhcos/b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/0 ignition.platform.id=openstack nohz=on nosoftlockup skew_tick=1 intel_pstate=disable intel_iommu=on iommu=pt default_hugepagesz=1G nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0 nosmt tsc=reliable rcu_nocbs=5-60 tuned.non_isolcpus=0000001f
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: xstate_offset[5]:  960, xstate_sizes[5]:   64
[    0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]:  512
[    0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[    0.000000] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]:    8
[    0.000000] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000557b8fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000557b9000-0x000000005d7c0fff] reserved
[    0.000000] BIOS-e820: [mem 0x000000005d7c1000-0x0000000060fd4fff] usable
[    0.000000] BIOS-e820: [mem 0x0000000060fd5000-0x0000000064fd4fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000064fd5000-0x000000006c5cefff] usable
[    0.000000] BIOS-e820: [mem 0x000000006c5cf000-0x000000006e7cefff] reserved
[    0.000000] BIOS-e820: [mem 0x000000006e7cf000-0x000000006f5fefff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000006f5ff000-0x000000006f7fefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000006f7ff000-0x000000006f7fffff] usable
[    0.000000] BIOS-e820: [mem 0x000000006f800000-0x000000008fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000603fffffff] usable
[    0.000000] process: using polling idle threads
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: Dell Inc. PowerEdge R640/08HT8T, BIOS 1.6.13 12/17/2018
[    0.000000] tsc: Detected 2000.000 MHz processor
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] last_pfn = 0x6040000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask 3FC000000000 write-back
[    0.000000]   1 base 004000000000 mask 3FE000000000 write-back
[    0.000000]   2 base 006000000000 mask 3FFFC0000000 write-back
[    0.000000]   3 base 000080000000 mask 3FFF80000000 uncachable
[    0.000000]   4 base 00007F000000 mask 3FFFFF000000 uncachable
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000]   8 disabled
[    0.000000]   9 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000000] total RAM covered: 392176M
[    0.000000]  gran_size: 64K  chunk_size: 64K     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 128K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 256K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 512K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 1M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 64K  chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 64K  chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 64K  chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 64K  chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 64K  chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 64K  chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 64K  chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 128K     chunk_size: 128K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 256K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 512K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 1M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 128K     chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 128K     chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 128K     chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 128K     chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 128K     chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 128K     chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 128K     chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 256K     chunk_size: 256K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 512K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 1M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 256K     chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 256K     chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 256K     chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 256K     chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 256K     chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 256K     chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 256K     chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 512K     chunk_size: 512K    num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 1M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 512K     chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 512K     chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 512K     chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 512K     chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 512K     chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 512K     chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 512K     chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 1M   chunk_size: 1M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 1M   chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 1M   chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 1M   chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 1M   chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 1M   chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 1M   chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 1M   chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 1M   chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 1M   chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 1M   chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 1M   chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 2M   chunk_size: 2M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 2M   chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 2M   chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 2M   chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 2M   chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 2M   chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 2M   chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 2M   chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 2M   chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 2M   chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 2M   chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 4M   chunk_size: 4M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 4M   chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 4M   chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 4M   chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 4M   chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 4M   chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 4M   chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 4M   chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 4M   chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 4M   chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 8M   chunk_size: 8M  num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 8M   chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 8M   chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 8M   chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 8M   chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 8M   chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 8M   chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 8M   chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 8M   chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 16M  chunk_size: 16M     num_reg: 10     lose cover RAM: 353G
[    0.000000]  gran_size: 16M  chunk_size: 32M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 16M  chunk_size: 64M     num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 16M  chunk_size: 128M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 16M  chunk_size: 256M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 16M  chunk_size: 512M    num_reg: 10     lose cover RAM: 0G
[    0.000000]  gran_size: 16M  chunk_size: 1G  num_reg: 10     lose cover RAM: 0G
[    0.000000] *BAD*gran_size: 16M  chunk_size: 2G  num_reg: 10     lose cover RAM: -1G
[    0.000000]  gran_size: 32M  chunk_size: 32M     num_reg: 10     lose cover RAM: 328720M
[    0.000000]  gran_size: 32M  chunk_size: 64M     num_reg: 10     lose cover RAM: 16M
[    0.000000]  gran_size: 32M  chunk_size: 128M    num_reg: 10     lose cover RAM: 16M
[    0.000000]  gran_size: 32M  chunk_size: 256M    num_reg: 10     lose cover RAM: 16M
[    0.000000]  gran_size: 32M  chunk_size: 512M    num_reg: 10     lose cover RAM: 16M
[    0.000000]  gran_size: 32M  chunk_size: 1G  num_reg: 10     lose cover RAM: 16M
[    0.000000] *BAD*gran_size: 32M  chunk_size: 2G  num_reg: 10     lose cover RAM: -1008M
[    0.000000]  gran_size: 64M  chunk_size: 64M     num_reg: 10     lose cover RAM: 263216M
[    0.000000]  gran_size: 64M  chunk_size: 128M    num_reg: 10     lose cover RAM: 48M
[    0.000000]  gran_size: 64M  chunk_size: 256M    num_reg: 10     lose cover RAM: 48M
[    0.000000]  gran_size: 64M  chunk_size: 512M    num_reg: 10     lose cover RAM: 48M
[    0.000000]  gran_size: 64M  chunk_size: 1G  num_reg: 10     lose cover RAM: 48M
[    0.000000] *BAD*gran_size: 64M  chunk_size: 2G  num_reg: 10     lose cover RAM: -976M
[    0.000000]  gran_size: 128M     chunk_size: 128M    num_reg: 10     lose cover RAM: 132208M
[    0.000000]  gran_size: 128M     chunk_size: 256M    num_reg: 10     lose cover RAM: 112M
[    0.000000]  gran_size: 128M     chunk_size: 512M    num_reg: 10     lose cover RAM: 112M
[    0.000000]  gran_size: 128M     chunk_size: 1G  num_reg: 10     lose cover RAM: 112M
[    0.000000] *BAD*gran_size: 128M     chunk_size: 2G  num_reg: 10     lose cover RAM: -912M
[    0.000000]  gran_size: 256M     chunk_size: 256M    num_reg: 10     lose cover RAM: 1264M
[    0.000000]  gran_size: 256M     chunk_size: 512M    num_reg: 10     lose cover RAM: 240M
[    0.000000]  gran_size: 256M     chunk_size: 1G  num_reg: 10     lose cover RAM: 240M
[    0.000000] *BAD*gran_size: 256M     chunk_size: 2G  num_reg: 10     lose cover RAM: -784M
[    0.000000]  gran_size: 512M     chunk_size: 512M    num_reg: 10     lose cover RAM: 496M
[    0.000000]  gran_size: 512M     chunk_size: 1G  num_reg: 10     lose cover RAM: 496M
[    0.000000] *BAD*gran_size: 512M     chunk_size: 2G  num_reg: 10     lose cover RAM: -528M
[    0.000000]  gran_size: 1G   chunk_size: 1G  num_reg: 9      lose cover RAM: 1008M
[    0.000000]  gran_size: 1G   chunk_size: 2G  num_reg: 10     lose cover RAM: 1008M
[    0.000000]  gran_size: 2G   chunk_size: 2G  num_reg: 7      lose cover RAM: 3056M
[    0.000000] mtrr_cleanup: can not find optimal value
[    0.000000] please specify mtrr_gran_size/mtrr_chunk_size
[    0.000000] e820: update [mem 0x7f000000-0xffffffff] usable ==> reserved
[    0.000000] last_pfn = 0x6f800 max_arch_pfn = 0x400000000
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x2c2b001000, 0x2c2b001fff] PGTABLE
[    0.000000] BRK [0x2c2b002000, 0x2c2b002fff] PGTABLE
[    0.000000] BRK [0x2c2b003000, 0x2c2b003fff] PGTABLE
[    0.000000] BRK [0x2c2b004000, 0x2c2b004fff] PGTABLE
[    0.000000] BRK [0x2c2b005000, 0x2c2b005fff] PGTABLE
[    0.000000] BRK [0x2c2b006000, 0x2c2b006fff] PGTABLE
[    0.000000] BRK [0x2c2b007000, 0x2c2b007fff] PGTABLE
[    0.000000] BRK [0x2c2b008000, 0x2c2b008fff] PGTABLE
[    0.000000] BRK [0x2c2b009000, 0x2c2b009fff] PGTABLE
[    0.000000] BRK [0x2c2b00a000, 0x2c2b00afff] PGTABLE
[    0.000000] BRK [0x2c2b00b000, 0x2c2b00bfff] PGTABLE
[    0.000000] BRK [0x2c2b00c000, 0x2c2b00cfff] PGTABLE
[    0.000000] RAMDISK: [mem 0x2f12c000-0x3388dfff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000FE320 000024 (v02 DELL  )
[    0.000000] ACPI: XSDT 0x000000006F6BF188 0000EC (v01 DELL   PE_SC3   00000000      01000013)
[    0.000000] ACPI: FACP 0x000000006F7F9000 000114 (v06 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: DSDT 0x000000006F79D000 033CDB (v02 DELL   PE_SC3   00000003 DELL 00000001)
[    0.000000] ACPI: FACS 0x000000006F0CF000 000040
[    0.000000] ACPI: MCEJ 0x000000006F7FC000 000130 (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: WD__ 0x000000006F7FB000 000134 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: SLIC 0x000000006F7FA000 000024 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: HPET 0x000000006F7F8000 000038 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: APIC 0x000000006F7F6000 0016DE (v04 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: MCFG 0x000000006F7F5000 00003C (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: MIGT 0x000000006F7F4000 000040 (v01 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: MSCT 0x000000006F7F3000 000090 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: PCAT 0x000000006F7D9000 000048 (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: PCCT 0x000000006F7D8000 00006E (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: RASF 0x000000006F7D7000 000030 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: SLIT 0x000000006F7D6000 00006C (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: SRAT 0x000000006F7D3000 002830 (v03 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: SVOS 0x000000006F7D2000 000032 (v01 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: WSMT 0x000000006F7D1000 000028 (v01 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: OEM4 0x000000006F6FA000 0A27C4 (v02 INTEL  CPU  CST 00003000 INTL 20150818)
[    0.000000] ACPI: SSDT 0x000000006F6C4000 035130 (v02 INTEL  SSDT  PM 00004000 INTL 20150818)
[    0.000000] ACPI: SSDT 0x000000006F6C3000 000761 (v02 DELL   PE_SC3   00000000 DELL 00000001)
[    0.000000] ACPI: SSDT 0x000000006F6C0000 002541 (v02 INTEL  SpsNm    00000002 INTL 20150818)
[    0.000000] ACPI: DMAR 0x000000006F7FD000 000260 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.000000] ACPI: HEST 0x000000006F7F2000 00017C (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: BERT 0x000000006F7F1000 000030 (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: ERST 0x000000006F7F0000 000230 (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: EINJ 0x000000006F7EF000 000150 (v01 DELL   PE_SC3   00000002 DELL 00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x40 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x48 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x42 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x46 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x44 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x18 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x58 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x10 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x50 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x16 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x56 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x12 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x52 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x14 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x54 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x20 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x60 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x28 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x68 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x22 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x62 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x26 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x66 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x24 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x64 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x38 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x78 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x30 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x70 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x36 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x76 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x32 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x72 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x34 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x74 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x41 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x49 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x43 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x47 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x45 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x19 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x59 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x11 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x51 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x17 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x57 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x13 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x53 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x15 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x55 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x21 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x61 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x29 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x69 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x23 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x63 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x27 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x67 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x25 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x65 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x39 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x79 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x31 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x71 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x37 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x77 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x33 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x73 -> Node 1
[    0.000000] SRAT: PXM 0 -> APIC 0x35 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x75 -> Node 1
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x303fffffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 1 [mem 0x3040000000-0x603fffffff]
[    0.000000] NUMA: Initialized distance table, cnt=2
[    0.000000] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x303fffffff] -> [mem 0x00000000-0x303fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x303ffd5000-0x303fffffff]
[    0.000000] NODE_DATA(1) allocated [mem 0x603ffd4000-0x603fffefff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000603fffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009bfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000557b8fff]
[    0.000000]   node   0: [mem 0x000000005d7c1000-0x0000000060fd4fff]
[    0.000000]   node   0: [mem 0x0000000064fd5000-0x000000006c5cefff]
[    0.000000]   node   0: [mem 0x000000006f7ff000-0x000000006f7fffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000303fffffff]
[    0.000000]   node   1: [mem 0x0000003040000000-0x000000603fffffff]
[    0.000000] Zeroed struct page in unavailable ranges: 64157 pages
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000303fffffff]
[    0.000000] On node 0 totalpages: 49939811
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3995 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 6104 pages used for memmap
[    0.000000]   DMA32 zone: 390600 pages, LIFO batch:31
[    0.000000]   Normal zone: 774144 pages used for memmap
[    0.000000]   Normal zone: 49545216 pages, LIFO batch:31
[    0.000000] Initmem setup node 1 [mem 0x0000003040000000-0x000000603fffffff]
[    0.000000] On node 1 totalpages: 50331648
[    0.000000]   Normal zone: 786432 pages used for memmap
[    0.000000]   Normal zone: 50331648 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x508
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: X2APIC_NMI (uid[0xffffffff] high level lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high level lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
[    0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-31
[    0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec08000, GSI 32-39
[    0.000000] IOAPIC[3]: apic_id 11, version 32, address 0xfec10000, GSI 40-47
[    0.000000] IOAPIC[4]: apic_id 12, version 32, address 0xfec18000, GSI 48-55
[    0.000000] IOAPIC[5]: apic_id 15, version 32, address 0xfec20000, GSI 72-79
[    0.000000] IOAPIC[6]: apic_id 16, version 32, address 0xfec28000, GSI 80-87
[    0.000000] IOAPIC[7]: apic_id 17, version 32, address 0xfec30000, GSI 88-95
[    0.000000] IOAPIC[8]: apic_id 18, version 32, address 0xfec38000, GSI 96-103
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[    0.000000] smpboot: Allowing 80 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0x557b9000-0x5d7c0fff]
[    0.000000] PM: Registered nosave memory: [mem 0x60fd5000-0x64fd4fff]
[    0.000000] PM: Registered nosave memory: [mem 0x6c5cf000-0x6e7cefff]
[    0.000000] PM: Registered nosave memory: [mem 0x6e7cf000-0x6f5fefff]
[    0.000000] PM: Registered nosave memory: [mem 0x6f5ff000-0x6f7fefff]
[    0.000000] PM: Registered nosave memory: [mem 0x6f800000-0x8fffffff]
[    0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfcffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfd000000-0xfe7fffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfe800000-0xfebfffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfec7ffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfec80000-0xfed00fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfed01000-0xfed3ffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfed40000-0xfed44fff]
[    0.000000] PM: Registered nosave memory: [mem 0xfed45000-0xfeffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xff000000-0xffffffff]
[    0.000000] [mem 0x90000000-0xfcffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8192 nr_cpu_ids:80 nr_node_ids:2
[    0.000000] percpu: Embedded 56 pages/cpu s192512 r8192 d28672 u262144
[    0.000000] pcpu-alloc: s192512 r8192 d28672 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 02 04 06 08 10 12 14 [0] 16 18 20 22 24 26 28 30 
[    0.000000] pcpu-alloc: [0] 32 34 36 38 40 42 44 46 [0] 48 50 52 54 56 58 60 62 
[    0.000000] pcpu-alloc: [0] 64 66 68 70 72 74 76 78 [1] 01 03 05 07 09 11 13 15 
[    0.000000] pcpu-alloc: [1] 17 19 21 23 25 27 29 31 [1] 33 35 37 39 41 43 45 47 
[    0.000000] pcpu-alloc: [1] 49 51 53 55 57 59 61 63 [1] 65 67 69 71 73 75 77 79 
[    0.000000] Built 2 zonelists, mobility grouping on.  Total pages: 98704694
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/vmlinuz-4.18.0-193.14.3.rt13.67.el8_2.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.0/rhcos/b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/0 ignition.platform.id=openstack nohz=on nosoftlockup skew_tick=1 intel_pstate=disable intel_iommu=on iommu=pt default_hugepagesz=1G nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0 nosmt tsc=reliable rcu_nocbs=5-60 tuned.non_isolcpus=0000001f
[    0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://access.redhat.com/ecosystem.
[    0.000000] DMAR: IOMMU enabled
[    0.000000] audit: disabled (until reboot)
[    0.000000] Memory: 1701808K/401085836K available (12293K kernel code, 2396K rwdata, 3792K rodata, 2328K init, 8416K bss, 6466072K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=80, Nodes=2
[    0.000000] Kernel/User page tables isolation: enabled
[    0.000000] ftrace: allocating 35608 entries in 140 pages
[    0.001000] rcu: Preemptible hierarchical RCU implementation.
[    0.001000] rcu:     RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=80.
[    0.001000] rcu:     RCU priority boosting: priority 1 delay 500 ms.
[    0.001000] rcu:     RCU_SOFTIRQ processing moved to rcuc kthreads.
[    0.001000]  No expedited grace period (rcu_normal_after_boot).
[    0.001000]  Tasks RCU enabled.
[    0.001000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.001000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=80
[    0.001000] NR_IRQS: 524544, nr_irqs: 2424, preallocated irqs: 16
[    0.001000] rcu:     Offload RCU callbacks from CPUs: 5-60.
[    0.001000] random: crng done (trusting CPU's manufacturer)
[    0.001000] Console: colour VGA+ 80x25
[    0.001000] printk: console [tty0] enabled
[    0.001000] printk: console [ttyS0] enabled
[    0.001000] ACPI: Core revision 20190703
[    0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[    0.001000] hpet clockevent registered
[    0.001006] APIC: Switch to symmetric I/O mode setup
[    0.001007] DMAR: Host address width 46
[    0.001008] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
[    0.001014] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001015] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[    0.001018] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001019] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0
[    0.001023] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001024] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[    0.001028] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001028] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0
[    0.001032] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001032] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0
[    0.001038] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001039] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[    0.001042] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001043] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1
[    0.001046] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[    0.001047] DMAR: RMRR base: 0x000000557b9000 end: 0x0000005d7c0fff
[    0.001048] DMAR: RMRR base: 0x0000006e2c9000 end: 0x0000006e7c8fff
[    0.001048] DMAR: RMRR base: 0x0000006f0c1000 end: 0x0000006f0c3fff
[    0.001049] DMAR: ATSR flags: 0x0
[    0.001049] DMAR: ATSR flags: 0x0
[    0.001051] DMAR-IR: IOAPIC id 12 under DRHD base  0xc5ffc000 IOMMU 6
[    0.001052] DMAR-IR: IOAPIC id 11 under DRHD base  0xb87fc000 IOMMU 5
[    0.001052] DMAR-IR: IOAPIC id 10 under DRHD base  0xaaffc000 IOMMU 4
[    0.001053] DMAR-IR: IOAPIC id 18 under DRHD base  0xfbffc000 IOMMU 3
[    0.001053] DMAR-IR: IOAPIC id 17 under DRHD base  0xee7fc000 IOMMU 2
[    0.001054] DMAR-IR: IOAPIC id 16 under DRHD base  0xe0ffc000 IOMMU 1
[    0.001055] DMAR-IR: IOAPIC id 15 under DRHD base  0xd37fc000 IOMMU 0
[    0.001055] DMAR-IR: IOAPIC id 8 under DRHD base  0x9d7fc000 IOMMU 7
[    0.001056] DMAR-IR: IOAPIC id 9 under DRHD base  0x9d7fc000 IOMMU 7
[    0.001057] DMAR-IR: HPET id 0 under DRHD base 0x9d7fc000
[    0.001058] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.001058] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.003085] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.003085] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.003092] Switched APIC routing to physical flat.
[    0.004000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.008013] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x39a85c9bff6, max_idle_ns: 881590591483 ns
[    0.008015] Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.00 BogoMIPS (lpj=2000000)
[    0.008018] pid_max: default: 81920 minimum: 640
[    0.008334] Security Framework initialized
[    0.008335] Yama: becoming mindful.
[    0.008339] SELinux:  Initializing.
[    0.063564] Dentry cache hash table entries: 16777216 (order: 16, 268435456 bytes)
[    0.112510] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes)
[    0.112981] Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.113381] Mountpoint-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.114090] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.114090] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.114105] process: WARNING: polling idle and HT enabled, performance may degrade
[    0.114107] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.114108] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.114109] FEATURE SPEC_CTRL Present
[    0.114109] FEATURE IBPB_SUPPORT Present
[    0.114110] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    0.114113] Spectre V2 : Using retpoline on Skylake-generation processors may not fully mitigate the vulnerability.
[    0.114113] Spectre V2 : Add the "spectre_v2=ibrs" kernel boot flag to enable IBRS on Skylake systems that need full mitigation.
[    0.114114] Spectre V2 : Mitigation: Full generic retpoline
[    0.114114] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.114114] Spectre V2 : Enabling Restricted Speculation for firmware calls
[    0.114116] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[    0.114116] Spectre V2 : User space: Mitigation: STIBP via seccomp and prctl
[    0.114118] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[    0.114119] TAA: Mitigation: Clear CPU buffers
[    0.114119] MDS: Mitigation: Clear CPU buffers
[    0.114445] Freeing SMP alternatives memory: 32K
[    0.116257] TSC deadline timer enabled
[    0.116269] smpboot: CPU0: Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz (family: 0x6, model: 0x55, stepping: 0x4)
[    0.611037] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[    0.621622] ... version:                4
[    0.625914] ... bit width:              48
[    0.630267] ... generic registers:      4
[    0.634562] ... value mask:             0000ffffffffffff
[    0.640168] ... max period:             00007fffffffffff
[    0.645761] ... fixed-purpose events:   3
[    0.650063] ... event mask:             000000070000000f
[    0.657003] rcu: Hierarchical SRCU implementation.
[    0.678003] smp: Bringing up secondary CPUs ...
[    0.696032] x86: Booting SMP configuration:
[    0.700501] .... node  #1, CPUs:        #1
[    0.001000] smpboot: CPU 1 Converting physical 0 to logical die 1
[    0.784027] .... node  #0, CPUs:    #2
[    0.818834] .... node  #1, CPUs:    #3
[    0.844068] .... node  #0, CPUs:    #4
[    0.868089] .... node  #1, CPUs:    #5
[    0.893082] .... node  #0, CPUs:    #6
[    0.917084] .... node  #1, CPUs:    #7
[    0.942075] .... node  #0, CPUs:    #8
[    0.966088] .... node  #1, CPUs:    #9
[    0.991078] .... node  #0, CPUs:   #10
[    1.015092] .... node  #1, CPUs:   #11
[    1.040077] .... node  #0, CPUs:   #12
[    1.064093] .... node  #1, CPUs:   #13
[    1.089074] .... node  #0, CPUs:   #14
[    1.113081] .... node  #1, CPUs:   #15
[    1.138074] .... node  #0, CPUs:   #16
[    1.162100] .... node  #1, CPUs:   #17
[    1.187086] .... node  #0, CPUs:   #18
[    1.211094] .... node  #1, CPUs:   #19
[    1.236073] .... node  #0, CPUs:   #20
[    1.261089] .... node  #1, CPUs:   #21
[    1.286077] .... node  #0, CPUs:   #22
[    1.310088] .... node  #1, CPUs:   #23
[    1.335082] .... node  #0, CPUs:   #24
[    1.359089] .... node  #1, CPUs:   #25
[    1.384075] .... node  #0, CPUs:   #26
[    1.408093] .... node  #1, CPUs:   #27
[    1.433076] .... node  #0, CPUs:   #28
[    1.457087] .... node  #1, CPUs:   #29
[    1.482082] .... node  #0, CPUs:   #30
[    1.507091] .... node  #1, CPUs:   #31
[    1.532074] .... node  #0, CPUs:   #32
[    1.556103] .... node  #1, CPUs:   #33
[    1.581075] .... node  #0, CPUs:   #34
[    1.605093] .... node  #1, CPUs:   #35
[    1.630074] .... node  #0, CPUs:   #36
[    1.654090] .... node  #1, CPUs:   #37
[    1.679080] .... node  #0, CPUs:   #38
[    1.703088] .... node  #1, CPUs:   #39
[    1.728080] .... node  #0, CPUs:   #40
[    1.749095] .... node  #1, CPUs:   #41
[    1.770065] .... node  #0, CPUs:   #42
[    1.793109] .... node  #1, CPUs:   #43
[    1.814065] .... node  #0, CPUs:   #44
[    1.835097] .... node  #1, CPUs:   #45
[    1.856070] .... node  #0, CPUs:   #46
[    1.883796] .... node  #1, CPUs:   #47
[    1.904068] .... node  #0, CPUs:   #48
[    1.925100] .... node  #1, CPUs:   #49
[    1.946074] .... node  #0, CPUs:   #50
[    1.967105] .... node  #1, CPUs:   #51
[    1.988069] .... node  #0, CPUs:   #52
[    2.011098] .... node  #1, CPUs:   #53
[    2.032061] .... node  #0, CPUs:   #54
[    2.053097] .... node  #1, CPUs:   #55
[    2.076071] .... node  #0, CPUs:   #56
[    2.097089] .... node  #1, CPUs:   #57
[    2.118066] .... node  #0, CPUs:   #58
[    2.139091] .... node  #1, CPUs:   #59
[    2.160077] .... node  #0, CPUs:   #60
[    2.181078] .... node  #1, CPUs:   #61
[    2.202045] .... node  #0, CPUs:   #62
[    2.223082] .... node  #1, CPUs:   #63
[    2.244056] .... node  #0, CPUs:   #64
[    2.267084] .... node  #1, CPUs:   #65
[    2.290058] .... node  #0, CPUs:   #66
[    2.311075] .... node  #1, CPUs:   #67
[    2.332055] .... node  #0, CPUs:   #68
[    2.353084] .... node  #1, CPUs:   #69
[    2.374059] .... node  #0, CPUs:   #70
[    2.397087] .... node  #1, CPUs:   #71
[    2.420058] .... node  #0, CPUs:   #72
[    2.443085] .... node  #1, CPUs:   #73
[    2.466045] .... node  #0, CPUs:   #74
[    2.489078] .... node  #1, CPUs:   #75
[    2.511058] .... node  #0, CPUs:   #76
[    2.532088] .... node  #1, CPUs:   #77
[    2.553059] .... node  #0, CPUs:   #78
[    2.575082] .... node  #1, CPUs:   #79
[    2.578076] smp: Brought up 2 nodes, 40 CPUs
[    2.586661] smpboot: Max logical packages: 4
[    2.591220] smpboot: Total of 40 processors activated (160101.32 BogoMIPS)
[    4.339171] node 0 initialised, 48686764 pages in 1737ms
[    4.361031] node 1 initialised, 49477189 pages in 1759ms
[    4.376600] devtmpfs: initialized
[    4.380370] x86/mm: Memory block size: 1024MB
[    4.389101] PM: Registering ACPI NVS region [mem 0x6e7cf000-0x6f5fefff] (14876672 bytes)
[    4.398291] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    4.408832] futex hash table entries: 32768 (order: 10, 4194304 bytes)
[    4.416742] pinctrl core: initialized pinctrl subsystem
[    4.422734] NET: Registered protocol family 16
[    4.427801] cpuidle: using governor menu
[    4.432058] Detected 1 PCC Subspaces
[    4.436008] Registering PCC driver as Mailbox controller
[    4.441663] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    4.450008] ACPI: bus type PCI registered
[    4.454248] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    4.461130] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[    4.471021] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[    4.478062] pmd_set_huge: Cannot satisfy [mem 0x80000000-0x80200000] with a huge-page mapping due to MTRR override.
[    4.489380] PCI: Using configuration type 1 for base access
[    4.495276] PCI: Dell System detected, enabling pci=bfsort.
[    4.505448] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    4.513123] cryptd: max_cpu_qlen set to 1000
[    4.518391] ACPI: Added _OSI(Module Device)
[    4.523077] ACPI: Added _OSI(Processor Device)
[    4.527805] ACPI: Added _OSI(3.0 _SCP Extensions)
[    4.532794] ACPI: Added _OSI(Processor Aggregator Device)
[    4.538471] ACPI: Added _OSI(Linux-Dell-Video)
[    4.543198] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    4.548789] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    4.612118] ACPI: 4 ACPI AML tables successfully acquired and loaded
[    4.624395] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[    4.636298] ACPI: Dynamic OEM Table Load:
[    4.777130] ACPI: Interpreter enabled
[    4.781112] ACPI: (supports S0 S5)
[    4.784808] ACPI: Using IOAPIC for interrupt routing
[    4.790155] HEST: Table parsing has been initialized.
[    4.795491] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    4.806048] ACPI: Enabled 5 GPEs in block 00 to 7F
[    4.865414] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-16])
[    4.871871] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    4.881442] acpi PNP0A08:00: PCIe AER handled by firmware
[    4.887158] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug LTR]
[    4.894945] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    4.903091] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[    4.911926] acpi PNP0A08:00: ignoring host bridge window [mem 0x000c4000-0x000cbfff window] (conflicts with Video ROM [mem 0x000c0000-0x000c7fff])
[    4.926830] PCI host bridge to bus 0000:00
[    4.931219] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    4.938273] pci_bus 0000:00: root bus resource [io  0x1000-0x3fff window]
[    4.945335] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    4.953303] pci_bus 0000:00: root bus resource [mem 0xfe010000-0xfe010fff window]
[    4.961282] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9d7fffff window]
[    4.969248] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x380fffffffff window]
[    4.978008] pci_bus 0000:00: root bus resource [bus 00-16]
[    4.983676] pci 0000:00:00.0: [8086:2020] type 00 class 0x060000
[    4.990613] pci 0000:00:05.0: [8086:2024] type 00 class 0x088000
[    4.997509] pci 0000:00:05.2: [8086:2025] type 00 class 0x088000
[    5.004409] pci 0000:00:05.4: [8086:2026] type 00 class 0x080020
[    5.010701] pci 0000:00:05.4: reg 0x10: [mem 0x92d20000-0x92d20fff]
[    5.017849] pci 0000:00:08.0: [8086:2014] type 00 class 0x088000
[    5.024726] pci 0000:00:08.1: [8086:2015] type 00 class 0x110100
[    5.031590] pci 0000:00:08.2: [8086:2016] type 00 class 0x088000
[    5.038482] pci 0000:00:11.0: [8086:a1ec] type 00 class 0xff0000
[    5.045437] pci 0000:00:11.5: [8086:a1d2] type 00 class 0x010601
[    5.051734] pci 0000:00:11.5: reg 0x10: [mem 0x92d16000-0x92d17fff]
[    5.058278] pci 0000:00:11.5: reg 0x14: [mem 0x92d1f000-0x92d1f0ff]
[    5.065008] pci 0000:00:11.5: reg 0x18: [io  0x2068-0x206f]
[    5.070678] pci 0000:00:11.5: reg 0x1c: [io  0x2074-0x2077]
[    5.076537] pci 0000:00:11.5: reg 0x20: [io  0x2040-0x205f]
[    5.082395] pci 0000:00:11.5: reg 0x24: [mem 0x92c80000-0x92cfffff]
[    5.089022] pci 0000:00:11.5: PME# supported from D3hot
[    5.095089] pci 0000:00:14.0: [8086:a1af] type 00 class 0x0c0330
[    5.101387] pci 0000:00:14.0: reg 0x10: [mem 0x92d00000-0x92d0ffff 64bit]
[    5.108504] pci 0000:00:14.0: PME# supported from D3hot D3cold
[    5.115228] pci 0000:00:14.2: [8086:a1b1] type 00 class 0x118000
[    5.121528] pci 0000:00:14.2: reg 0x10: [mem 0x92d1c000-0x92d1cfff 64bit]
[    5.129241] pci 0000:00:16.0: [8086:a1ba] type 00 class 0x078000
[    5.135550] pci 0000:00:16.0: reg 0x10: [mem 0x92d1b000-0x92d1bfff 64bit]
[    5.142690] pci 0000:00:16.0: PME# supported from D3hot
[    5.148789] pci 0000:00:16.1: [8086:a1bb] type 00 class 0x078000
[    5.155095] pci 0000:00:16.1: reg 0x10: [mem 0x92d1a000-0x92d1afff 64bit]
[    5.162252] pci 0000:00:16.1: PME# supported from D3hot
[    5.168350] pci 0000:00:16.4: [8086:a1be] type 00 class 0x078000
[    5.174655] pci 0000:00:16.4: reg 0x10: [mem 0x92d19000-0x92d19fff 64bit]
[    5.182023] pci 0000:00:16.4: PME# supported from D3hot
[    5.187903] pci 0000:00:17.0: [8086:a182] type 00 class 0x010601
[    5.194201] pci 0000:00:17.0: reg 0x10: [mem 0x92d14000-0x92d15fff]
[    5.200741] pci 0000:00:17.0: reg 0x14: [mem 0x92d1e000-0x92d1e0ff]
[    5.207277] pci 0000:00:17.0: reg 0x18: [io  0x2060-0x2067]
[    5.213130] pci 0000:00:17.0: reg 0x1c: [io  0x2070-0x2073]
[    5.219022] pci 0000:00:17.0: reg 0x20: [io  0x2020-0x203f]
[    5.225022] pci 0000:00:17.0: reg 0x24: [mem 0x92c00000-0x92c7ffff]
[    5.231418] pci 0000:00:17.0: PME# supported from D3hot
[    5.237542] pci 0000:00:1c.0: [8086:a190] type 01 class 0x060400
[    5.244442] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[    5.251455] pci 0000:00:1c.4: [8086:a194] type 01 class 0x060400
[    5.257813] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[    5.264839] pci 0000:00:1f.0: [8086:a1c1] type 00 class 0x060100
[    5.271839] pci 0000:00:1f.2: [8086:a1a1] type 00 class 0x058000
[    5.278141] pci 0000:00:1f.2: reg 0x10: [mem 0x92d10000-0x92d13fff]
[    5.285334] pci 0000:00:1f.4: [8086:a1a3] type 00 class 0x0c0500
[    5.292023] pci 0000:00:1f.4: reg 0x10: [mem 0x92d18000-0x92d180ff 64bit]
[    5.299022] pci 0000:00:1f.4: reg 0x20: [io  0x2000-0x201f]
[    5.305180] pci 0000:00:1f.5: [8086:a1a4] type 00 class 0x0c8000
[    5.311482] pci 0000:00:1f.5: reg 0x10: [mem 0xfe010000-0xfe010fff]
[    5.318682] pci 0000:00:1c.0: PCI bridge to [bus 01]
[    5.323935] pci 0000:00:1c.0:   bridge window [mem 0x92a00000-0x92bfffff]
[    5.331049] pci 0000:02:00.0: [1556:be00] type 01 class 0x060400
[    5.341014] pci 0000:00:1c.4: PCI bridge to [bus 02-03]
[    5.347025] pci 0000:00:1c.4:   bridge window [mem 0x92000000-0x928fffff]
[    5.354023] pci 0000:00:1c.4:   bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[    5.362022] pci_bus 0000:03: extended config space not accessible
[    5.368256] pci 0000:03:00.0: [102b:0536] type 00 class 0x030000
[    5.375022] pci 0000:03:00.0: reg 0x10: [mem 0x91000000-0x91ffffff pref]
[    5.382023] pci 0000:03:00.0: reg 0x14: [mem 0x92808000-0x9280bfff]
[    5.388094] pci 0000:03:00.0: reg 0x18: [mem 0x92000000-0x927fffff]
[    5.394790] pci 0000:02:00.0: PCI bridge to [bus 03]
[    5.400047] pci 0000:02:00.0:   bridge window [mem 0x92000000-0x928fffff]
[    5.407115] pci 0000:02:00.0:   bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[    5.415355] pci_bus 0000:00: on NUMA node 0
[    5.415651] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15)
[    5.423022] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 *6 10 11 12 14 15)
[    5.430070] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15)
[    5.437286] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15)
[    5.445022] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 *11 12 14 15)
[    5.452022] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 *6 10 11 12 14 15)
[    5.459022] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 *5 6 10 11 12 14 15)
[    5.466145] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 10 *11 12 14 15)
[    5.473586] ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 17-39])
[    5.486022] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    5.495263] acpi PNP0A08:01: PCIe AER handled by firmware
[    5.501021] acpi PNP0A08:01: _OSC: platform does not support [SHPCHotplug LTR]
[    5.509022] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    5.517022] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
[    5.526024] PCI host bridge to bus 0000:17
[    5.530230] pci_bus 0000:17: root bus resource [io  0x4000-0x5fff window]
[    5.537273] pci_bus 0000:17: root bus resource [mem 0x9d800000-0xaaffffff window]
[    5.545249] pci_bus 0000:17: root bus resource [mem 0x381000000000-0x381fffffffff window]
[    5.554016] pci_bus 0000:17: root bus resource [bus 17-39]
[    5.560011] pci 0000:17:00.0: [8086:2030] type 01 class 0x060400
[    5.566061] pci 0000:17:00.0: PME# supported from D0 D3hot D3cold
[    5.572548] pci 0000:17:02.0: [8086:2032] type 01 class 0x060400
[    5.579016] pci 0000:17:02.0: PME# supported from D0 D3hot D3cold
[    5.585350] pci 0000:17:05.0: [8086:2034] type 00 class 0x088000
[    5.592010] pci 0000:17:05.2: [8086:2035] type 00 class 0x088000
[    5.598116] pci 0000:17:05.4: [8086:2036] type 00 class 0x080020
[    5.605010] pci 0000:17:05.4: reg 0x10: [mem 0xa2c00000-0xa2c00fff]
[    5.611068] pci 0000:17:08.0: [8086:208d] type 00 class 0x088000
[    5.618010] pci 0000:17:08.1: [8086:208d] type 00 class 0x088000
[    5.624010] pci 0000:17:08.2: [8086:208d] type 00 class 0x088000
[    5.630193] pci 0000:17:08.3: [8086:208d] type 00 class 0x088000
[    5.637010] pci 0000:17:08.4: [8086:208d] type 00 class 0x088000
[    5.643016] pci 0000:17:08.5: [8086:208d] type 00 class 0x088000
[    5.649321] pci 0000:17:08.6: [8086:208d] type 00 class 0x088000
[    5.656010] pci 0000:17:08.7: [8086:208d] type 00 class 0x088000
[    5.662072] pci 0000:17:09.0: [8086:208d] type 00 class 0x088000
[    5.669009] pci 0000:17:09.1: [8086:208d] type 00 class 0x088000
[    5.675016] pci 0000:17:09.2: [8086:208d] type 00 class 0x088000
[    5.681195] pci 0000:17:09.3: [8086:208d] type 00 class 0x088000
[    5.687562] pci 0000:17:09.4: [8086:208d] type 00 class 0x088000
[    5.693935] pci 0000:17:09.5: [8086:208d] type 00 class 0x088000
[    5.700305] pci 0000:17:09.6: [8086:208d] type 00 class 0x088000
[    5.707010] pci 0000:17:09.7: [8086:208d] type 00 class 0x088000
[    5.713049] pci 0000:17:0a.0: [8086:208d] type 00 class 0x088000
[    5.720010] pci 0000:17:0a.1: [8086:208d] type 00 class 0x088000
[    5.726017] pci 0000:17:0a.2: [8086:208d] type 00 class 0x088000
[    5.732159] pci 0000:17:0a.3: [8086:208d] type 00 class 0x088000
[    5.739010] pci 0000:17:0a.4: [8086:208d] type 00 class 0x088000
[    5.745015] pci 0000:17:0a.5: [8086:208d] type 00 class 0x088000
[    5.751279] pci 0000:17:0a.6: [8086:208d] type 00 class 0x088000
[    5.758010] pci 0000:17:0a.7: [8086:208d] type 00 class 0x088000
[    5.764016] pci 0000:17:0b.0: [8086:208d] type 00 class 0x088000
[    5.771010] pci 0000:17:0b.1: [8086:208d] type 00 class 0x088000
[    5.777017] pci 0000:17:0b.2: [8086:208d] type 00 class 0x088000
[    5.783090] pci 0000:17:0b.3: [8086:208d] type 00 class 0x088000
[    5.790010] pci 0000:17:0e.0: [8086:208e] type 00 class 0x088000
[    5.796017] pci 0000:17:0e.1: [8086:208e] type 00 class 0x088000
[    5.802211] pci 0000:17:0e.2: [8086:208e] type 00 class 0x088000
[    5.809009] pci 0000:17:0e.3: [8086:208e] type 00 class 0x088000
[    5.814946] pci 0000:17:0e.4: [8086:208e] type 00 class 0x088000
[    5.821321] pci 0000:17:0e.5: [8086:208e] type 00 class 0x088000
[    5.828017] pci 0000:17:0e.6: [8086:208e] type 00 class 0x088000
[    5.834062] pci 0000:17:0e.7: [8086:208e] type 00 class 0x088000
[    5.841011] pci 0000:17:0f.0: [8086:208e] type 00 class 0x088000
[    5.847027] pci 0000:17:0f.1: [8086:208e] type 00 class 0x088000
[    5.853181] pci 0000:17:0f.2: [8086:208e] type 00 class 0x088000
[    5.860010] pci 0000:17:0f.3: [8086:208e] type 00 class 0x088000
[    5.866017] pci 0000:17:0f.4: [8086:208e] type 00 class 0x088000
[    5.872298] pci 0000:17:0f.5: [8086:208e] type 00 class 0x088000
[    5.879016] pci 0000:17:0f.6: [8086:208e] type 00 class 0x088000
[    5.885061] pci 0000:17:0f.7: [8086:208e] type 00 class 0x088000
[    5.892010] pci 0000:17:10.0: [8086:208e] type 00 class 0x088000
[    5.898016] pci 0000:17:10.1: [8086:208e] type 00 class 0x088000
[    5.904173] pci 0000:17:10.2: [8086:208e] type 00 class 0x088000
[    5.911010] pci 0000:17:10.3: [8086:208e] type 00 class 0x088000
[    5.917016] pci 0000:17:10.4: [8086:208e] type 00 class 0x088000
[    5.923282] pci 0000:17:10.5: [8086:208e] type 00 class 0x088000
[    5.930016] pci 0000:17:10.6: [8086:208e] type 00 class 0x088000
[    5.936025] pci 0000:17:10.7: [8086:208e] type 00 class 0x088000
[    5.943010] pci 0000:17:11.0: [8086:208e] type 00 class 0x088000
[    5.949021] pci 0000:17:11.1: [8086:208e] type 00 class 0x088000
[    5.955089] pci 0000:17:11.2: [8086:208e] type 00 class 0x088000
[    5.962016] pci 0000:17:11.3: [8086:208e] type 00 class 0x088000
[    5.968016] pci 0000:17:1d.0: [8086:2054] type 00 class 0x088000
[    5.974207] pci 0000:17:1d.1: [8086:2055] type 00 class 0x088000
[    5.981016] pci 0000:17:1d.2: [8086:2056] type 00 class 0x088000
[    5.987026] pci 0000:17:1d.3: [8086:2057] type 00 class 0x088000
[    5.994010] pci 0000:17:1e.0: [8086:2080] type 00 class 0x088000
[    6.000023] pci 0000:17:1e.1: [8086:2081] type 00 class 0x088000
[    6.006067] pci 0000:17:1e.2: [8086:2082] type 00 class 0x088000
[    6.012435] pci 0000:17:1e.3: [8086:2083] type 00 class 0x088000
[    6.019017] pci 0000:17:1e.4: [8086:2084] type 00 class 0x088000
[    6.025169] pci 0000:17:1e.5: [8086:2085] type 00 class 0x088000
[    6.032024] pci 0000:17:1e.6: [8086:2086] type 00 class 0x088000
[    6.038016] pci 0000:18:00.0: [1000:005f] type 00 class 0x010400
[    6.045010] pci 0000:18:00.0: reg 0x10: [io  0x4000-0x40ff]
[    6.050086] pci 0000:18:00.0: reg 0x14: [mem 0xa2b00000-0xa2b0ffff 64bit]
[    6.058010] pci 0000:18:00.0: reg 0x1c: [mem 0xa2a00000-0xa2afffff 64bit]
[    6.065010] pci 0000:18:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    6.072011] pci 0000:18:00.0: supports D1 D2
[    6.076022] pci 0000:17:00.0: PCI bridge to [bus 18]
[    6.082010] pci 0000:17:00.0:   bridge window [io  0x4000-0x4fff]
[    6.087501] pci 0000:17:00.0:   bridge window [mem 0xa2a00000-0xa2bfffff]
[    6.095016] pci 0000:19:00.0: [15b3:1015] type 00 class 0x020000
[    6.102016] pci 0000:19:00.0: reg 0x10: [mem 0xa0000000-0xa1ffffff 64bit pref]
[    6.110010] pci 0000:19:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    6.117064] pci 0000:19:00.0: PME# supported from D3cold
[    6.123016] pci 0000:19:00.0: reg 0x1a4: [mem 0xa2500000-0xa25fffff 64bit pref]
[    6.131016] pci 0000:19:00.0: VF(n) BAR0 space: [mem 0xa2500000-0xa29fffff 64bit pref] (contains BAR0 for 5 VFs)
[    6.143182] pci 0000:19:00.1: [15b3:1015] type 00 class 0x020000
[    6.149651] pci 0000:19:00.1: reg 0x10: [mem 0x9e000000-0x9fffffff 64bit pref]
[    6.157636] pci 0000:19:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    6.165273] pci 0000:19:00.1: PME# supported from D3cold
[    6.171091] pci 0000:19:00.1: reg 0x1a4: [mem 0xa2000000-0xa20fffff 64bit pref]
[    6.178898] pci 0000:19:00.1: VF(n) BAR0 space: [mem 0xa2000000-0xa24fffff 64bit pref] (contains BAR0 for 5 VFs)
[    6.190294] pci 0000:17:02.0: PCI bridge to [bus 19-1a]
[    6.195807] pci 0000:17:02.0:   bridge window [mem 0x9e000000-0xa29fffff 64bit pref]
[    6.204054] pci_bus 0000:17: on NUMA node 0
[    6.204179] ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 3a-5c])
[    6.210637] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    6.220206] acpi PNP0A08:02: PCIe AER handled by firmware
[    6.225940] acpi PNP0A08:02: _OSC: platform does not support [SHPCHotplug LTR]
[    6.233763] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    6.241906] acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration
[    6.250742] PCI host bridge to bus 0000:3a
[    6.255127] pci_bus 0000:3a: root bus resource [io  0x6000-0x7fff window]
[    6.262193] pci_bus 0000:3a: root bus resource [mem 0xab000000-0xb87fffff window]
[    6.270171] pci_bus 0000:3a: root bus resource [mem 0x382000000000-0x382fffffffff window]
[    6.278842] pci_bus 0000:3a: root bus resource [bus 3a-5c]
[    6.284620] pci 0000:3a:00.0: [8086:2030] type 01 class 0x060400
[    6.290996] pci 0000:3a:00.0: PME# supported from D0 D3hot D3cold
[    6.297485] pci 0000:3a:05.0: [8086:2034] type 00 class 0x088000
[    6.303874] pci 0000:3a:05.2: [8086:2035] type 00 class 0x088000
[    6.310270] pci 0000:3a:05.4: [8086:2036] type 00 class 0x080020
[    6.316558] pci 0000:3a:05.4: reg 0x10: [mem 0xb0000000-0xb0000fff]
[    6.323211] pci 0000:3a:08.0: [8086:2066] type 00 class 0x088000
[    6.329594] pci 0000:3a:09.0: [8086:2066] type 00 class 0x088000
[    6.335984] pci 0000:3a:0a.0: [8086:2040] type 00 class 0x088000
[    6.342363] pci 0000:3a:0a.1: [8086:2041] type 00 class 0x088000
[    6.348745] pci 0000:3a:0a.2: [8086:2042] type 00 class 0x088000
[    6.355097] pci 0000:3a:0a.3: [8086:2043] type 00 class 0x088000
[    6.361481] pci 0000:3a:0a.4: [8086:2044] type 00 class 0x088000
[    6.367862] pci 0000:3a:0a.5: [8086:2045] type 00 class 0x088000
[    6.374239] pci 0000:3a:0a.6: [8086:2046] type 00 class 0x088000
[    6.380623] pci 0000:3a:0a.7: [8086:2047] type 00 class 0x088000
[    6.387008] pci 0000:3a:0b.0: [8086:2048] type 00 class 0x088000
[    6.393380] pci 0000:3a:0b.1: [8086:2049] type 00 class 0x088000
[    6.399759] pci 0000:3a:0b.2: [8086:204a] type 00 class 0x088000
[    6.406132] pci 0000:3a:0b.3: [8086:204b] type 00 class 0x088000
[    6.412513] pci 0000:3a:0c.0: [8086:2040] type 00 class 0x088000
[    6.418903] pci 0000:3a:0c.1: [8086:2041] type 00 class 0x088000
[    6.425283] pci 0000:3a:0c.2: [8086:2042] type 00 class 0x088000
[    6.431660] pci 0000:3a:0c.3: [8086:2043] type 00 class 0x088000
[    6.438036] pci 0000:3a:0c.4: [8086:2044] type 00 class 0x088000
[    6.444412] pci 0000:3a:0c.5: [8086:2045] type 00 class 0x088000
[    6.450794] pci 0000:3a:0c.6: [8086:2046] type 00 class 0x088000
[    6.457173] pci 0000:3a:0c.7: [8086:2047] type 00 class 0x088000
[    6.463549] pci 0000:3a:0d.0: [8086:2048] type 00 class 0x088000
[    6.469927] pci 0000:3a:0d.1: [8086:2049] type 00 class 0x088000
[    6.476308] pci 0000:3a:0d.2: [8086:204a] type 00 class 0x088000
[    6.482684] pci 0000:3a:0d.3: [8086:204b] type 00 class 0x088000
[    6.489174] pci 0000:3b:00.0: [15b3:1015] type 00 class 0x020000
[    6.495645] pci 0000:3b:00.0: reg 0x10: [mem 0xae000000-0xafffffff 64bit pref]
[    6.503630] pci 0000:3b:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    6.511270] pci 0000:3b:00.0: PME# supported from D3cold
[    6.517213] pci 0000:3b:00.1: [15b3:1015] type 00 class 0x020000
[    6.523672] pci 0000:3b:00.1: reg 0x10: [mem 0xac000000-0xadffffff 64bit pref]
[    6.531655] pci 0000:3b:00.1: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[    6.539236] pci 0000:3b:00.1: PME# supported from D3cold
[    6.550779] pci 0000:3a:00.0: PCI bridge to [bus 3b]
[    6.556008] pci 0000:3a:00.0:   bridge window [mem 0xac000000-0xafffffff 64bit pref]
[    6.564248] pci_bus 0000:3a: on NUMA node 0
[    6.564338] ACPI: PCI Root Bridge [PC03] (domain 0000 [bus 5d-7f])
[    6.571008] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    6.580372] acpi PNP0A08:03: PCIe AER handled by firmware
[    6.586098] acpi PNP0A08:03: _OSC: platform does not support [SHPCHotplug LTR]
[    6.593913] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    6.602064] acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration
[    6.610931] PCI host bridge to bus 0000:5d
[    6.615311] pci_bus 0000:5d: root bus resource [io  0x8000-0x9fff window]
[    6.622382] pci_bus 0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]
[    6.630370] pci_bus 0000:5d: root bus resource [mem 0x383000000000-0x383fffffffff window]
[    6.639008] pci_bus 0000:5d: root bus resource [bus 5d-7f]
[    6.645008] pci 0000:5d:05.0: [8086:2034] type 00 class 0x088000
[    6.651176] pci 0000:5d:05.2: [8086:2035] type 00 class 0x088000
[    6.657572] pci 0000:5d:05.4: [8086:2036] type 00 class 0x080020
[    6.664008] pci 0000:5d:05.4: reg 0x10: [mem 0xb8800000-0xb8800fff]
[    6.670514] pci 0000:5d:0e.0: [8086:2058] type 00 class 0x110100
[    6.676895] pci 0000:5d:0e.1: [8086:2059] type 00 class 0x088000
[    6.683273] pci 0000:5d:0f.0: [8086:2058] type 00 class 0x110100
[    6.689649] pci 0000:5d:0f.1: [8086:2059] type 00 class 0x088000
[    6.696025] pci 0000:5d:10.0: [8086:2058] type 00 class 0x110100
[    6.702406] pci 0000:5d:10.1: [8086:2059] type 00 class 0x088000
[    6.708798] pci 0000:5d:12.0: [8086:204c] type 00 class 0x110100
[    6.715175] pci 0000:5d:12.1: [8086:204d] type 00 class 0x110100
[    6.721550] pci 0000:5d:12.2: [8086:204e] type 00 class 0x088000
[    6.728008] pci 0000:5d:12.4: [8086:204c] type 00 class 0x110100
[    6.734288] pci 0000:5d:12.5: [8086:204d] type 00 class 0x110100
[    6.740659] pci 0000:5d:15.0: [8086:2018] type 00 class 0x088000
[    6.747027] pci 0000:5d:16.0: [8086:2018] type 00 class 0x088000
[    6.753403] pci 0000:5d:16.4: [8086:2018] type 00 class 0x088000
[    6.760008] pci 0000:5d:17.0: [8086:2018] type 00 class 0x088000
[    6.766145] pci_bus 0000:5d: on NUMA node 0
[    6.766303] ACPI: PCI Root Bridge [PC06] (domain 0000 [bus 80-84])
[    6.773008] acpi PNP0A08:06: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    6.782341] acpi PNP0A08:06: PCIe AER handled by firmware
[    6.788060] acpi PNP0A08:06: _OSC: platform does not support [SHPCHotplug LTR]
[    6.796014] acpi PNP0A08:06: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    6.804008] acpi PNP0A08:06: FADT indicates ASPM is unsupported, using BIOS configuration
[    6.813008] acpi PNP0A08:06: host bridge window [io  0x0000 window] (ignored, not CPU addressable)
[    6.822267] PCI host bridge to bus 0000:80
[    6.827008] pci_bus 0000:80: root bus resource [mem 0xc6000000-0xd37fffff window]
[    6.834628] pci_bus 0000:80: root bus resource [mem 0x384000000000-0x384fffffffff window]
[    6.843299] pci_bus 0000:80: root bus resource [bus 80-84]
[    6.849083] pci 0000:80:05.0: [8086:2024] type 00 class 0x088000
[    6.855462] pci 0000:80:05.2: [8086:2025] type 00 class 0x088000
[    6.862008] pci 0000:80:05.4: [8086:2026] type 00 class 0x080020
[    6.868127] pci 0000:80:05.4: reg 0x10: [mem 0xc6000000-0xc6000fff]
[    6.875008] pci 0000:80:08.0: [8086:2014] type 00 class 0x088000
[    6.881078] pci 0000:80:08.1: [8086:2015] type 00 class 0x110100
[    6.887425] pci 0000:80:08.2: [8086:2016] type 00 class 0x088000
[    6.894008] pci_bus 0000:80: on NUMA node 1
[    6.894008] ACPI: PCI Root Bridge [PC07] (domain 0000 [bus 85-ad])
[    6.900314] acpi PNP0A08:07: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    6.910008] acpi PNP0A08:07: PCIe AER handled by firmware
[    6.916007] acpi PNP0A08:07: _OSC: platform does not support [SHPCHotplug LTR]
[    6.923459] acpi PNP0A08:07: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    6.931608] acpi PNP0A08:07: FADT indicates ASPM is unsupported, using BIOS configuration
[    6.940296] PCI host bridge to bus 0000:85
[    6.944685] pci_bus 0000:85: root bus resource [io  0xa000-0xbfff window]
[    6.952016] pci_bus 0000:85: root bus resource [mem 0xd3800000-0xe0ffffff window]
[    6.960008] pci_bus 0000:85: root bus resource [mem 0x385000000000-0x385fffffffff window]
[    6.969008] pci_bus 0000:85: root bus resource [bus 85-ad]
[    6.974155] pci 0000:85:00.0: [8086:2030] type 01 class 0x060400
[    6.981008] pci 0000:85:00.0: PME# supported from D0 D3hot D3cold
[    6.987057] pci 0000:85:01.0: [8086:2031] type 01 class 0x060400
[    6.994008] pci 0000:85:01.0: PME# supported from D0 D3hot D3cold
[    7.000024] pci 0000:85:02.0: [8086:2032] type 01 class 0x060400
[    7.007008] pci 0000:85:02.0: PME# supported from D0 D3hot D3cold
[    7.013008] pci 0000:85:03.0: [8086:2033] type 01 class 0x060400
[    7.019191] pci 0000:85:03.0: PME# supported from D0 D3hot D3cold
[    7.026008] pci 0000:85:05.0: [8086:2034] type 00 class 0x088000
[    7.032042] pci 0000:85:05.2: [8086:2035] type 00 class 0x088000
[    7.039008] pci 0000:85:05.4: [8086:2036] type 00 class 0x080020
[    7.045024] pci 0000:85:05.4: reg 0x10: [mem 0xd3c00000-0xd3c00fff]
[    7.051401] pci 0000:85:08.0: [8086:208d] type 00 class 0x088000
[    7.058023] pci 0000:85:08.1: [8086:208d] type 00 class 0x088000
[    7.064168] pci 0000:85:08.2: [8086:208d] type 00 class 0x088000
[    7.071008] pci 0000:85:08.3: [8086:208d] type 00 class 0x088000
[    7.077024] pci 0000:85:08.4: [8086:208d] type 00 class 0x088000
[    7.083286] pci 0000:85:08.5: [8086:208d] type 00 class 0x088000
[    7.090007] pci 0000:85:08.6: [8086:208d] type 00 class 0x088000
[    7.096039] pci 0000:85:08.7: [8086:208d] type 00 class 0x088000
[    7.103008] pci 0000:85:09.0: [8086:208d] type 00 class 0x088000
[    7.109022] pci 0000:85:09.1: [8086:208d] type 00 class 0x088000
[    7.115174] pci 0000:85:09.2: [8086:208d] type 00 class 0x088000
[    7.122008] pci 0000:85:09.3: [8086:208d] type 00 class 0x088000
[    7.128023] pci 0000:85:09.4: [8086:208d] type 00 class 0x088000
[    7.135007] pci 0000:85:09.5: [8086:208d] type 00 class 0x088000
[    7.141008] pci 0000:85:09.6: [8086:208d] type 00 class 0x088000
[    7.147038] pci 0000:85:09.7: [8086:208d] type 00 class 0x088000
[    7.154007] pci 0000:85:0a.0: [8086:208d] type 00 class 0x088000
[    7.160023] pci 0000:85:0a.1: [8086:208d] type 00 class 0x088000
[    7.166165] pci 0000:85:0a.2: [8086:208d] type 00 class 0x088000
[    7.173008] pci 0000:85:0a.3: [8086:208d] type 00 class 0x088000
[    7.179023] pci 0000:85:0a.4: [8086:208d] type 00 class 0x088000
[    7.186008] pci 0000:85:0a.5: [8086:208d] type 00 class 0x088000
[    7.192027] pci 0000:85:0a.6: [8086:208d] type 00 class 0x088000
[    7.198065] pci 0000:85:0a.7: [8086:208d] type 00 class 0x088000
[    7.205008] pci 0000:85:0b.0: [8086:208d] type 00 class 0x088000
[    7.211025] pci 0000:85:0b.1: [8086:208d] type 00 class 0x088000
[    7.218014] pci 0000:85:0b.2: [8086:208d] type 00 class 0x088000
[    7.224009] pci 0000:85:0b.3: [8086:208d] type 00 class 0x088000
[    7.230023] pci 0000:85:0e.0: [8086:208e] type 00 class 0x088000
[    7.236342] pci 0000:85:0e.1: [8086:208e] type 00 class 0x088000
[    7.243021] pci 0000:85:0e.2: [8086:208e] type 00 class 0x088000
[    7.249083] pci 0000:85:0e.3: [8086:208e] type 00 class 0x088000
[    7.256008] pci 0000:85:0e.4: [8086:208e] type 00 class 0x088000
[    7.262023] pci 0000:85:0e.5: [8086:208e] type 00 class 0x088000
[    7.268204] pci 0000:85:0e.6: [8086:208e] type 00 class 0x088000
[    7.275008] pci 0000:85:0e.7: [8086:208e] type 00 class 0x088000
[    7.281022] pci 0000:85:0f.0: [8086:208e] type 00 class 0x088000
[    7.288008] pci 0000:85:0f.1: [8086:208e] type 00 class 0x088000
[    7.294022] pci 0000:85:0f.2: [8086:208e] type 00 class 0x088000
[    7.300075] pci 0000:85:0f.3: [8086:208e] type 00 class 0x088000
[    7.307008] pci 0000:85:0f.4: [8086:208e] type 00 class 0x088000
[    7.312822] pci 0000:85:0f.5: [8086:208e] type 00 class 0x088000
[    7.320015] pci 0000:85:0f.6: [8086:208e] type 00 class 0x088000
[    7.326008] pci 0000:85:0f.7: [8086:208e] type 00 class 0x088000
[    7.332023] pci 0000:85:10.0: [8086:208e] type 00 class 0x088000
[    7.339009] pci 0000:85:10.1: [8086:208e] type 00 class 0x088000
[    7.345023] pci 0000:85:10.2: [8086:208e] type 00 class 0x088000
[    7.351091] pci 0000:85:10.3: [8086:208e] type 00 class 0x088000
[    7.358008] pci 0000:85:10.4: [8086:208e] type 00 class 0x088000
[    7.364024] pci 0000:85:10.5: [8086:208e] type 00 class 0x088000
[    7.371008] pci 0000:85:10.6: [8086:208e] type 00 class 0x088000
[    7.376589] pci 0000:85:10.7: [8086:208e] type 00 class 0x088000
[    7.383023] pci 0000:85:11.0: [8086:208e] type 00 class 0x088000
[    7.390007] pci 0000:85:11.1: [8086:208e] type 00 class 0x088000
[    7.396024] pci 0000:85:11.2: [8086:208e] type 00 class 0x088000
[    7.402084] pci 0000:85:11.3: [8086:208e] type 00 class 0x088000
[    7.409008] pci 0000:85:1d.0: [8086:2054] type 00 class 0x088000
[    7.415023] pci 0000:85:1d.1: [8086:2055] type 00 class 0x088000
[    7.422008] pci 0000:85:1d.2: [8086:2056] type 00 class 0x088000
[    7.428022] pci 0000:85:1d.3: [8086:2057] type 00 class 0x088000
[    7.434023] pci 0000:85:1e.0: [8086:2080] type 00 class 0x088000
[    7.441008] pci 0000:85:1e.1: [8086:2081] type 00 class 0x088000
[    7.447023] pci 0000:85:1e.2: [8086:2082] type 00 class 0x088000
[    7.454090] pci 0000:85:1e.3: [8086:2083] type 00 class 0x088000
[    7.460468] pci 0000:85:1e.4: [8086:2084] type 00 class 0x088000
[    7.466844] pci 0000:85:1e.5: [8086:2085] type 00 class 0x088000
[    7.473216] pci 0000:85:1e.6: [8086:2086] type 00 class 0x088000
[    7.479633] pci 0000:86:00.0: [8086:0a54] type 00 class 0x010802
[    7.485933] pci 0000:86:00.0: reg 0x10: [mem 0xd3b00000-0xd3b03fff 64bit]
[    7.493202] pci 0000:85:00.0: PCI bridge to [bus 86]
[    7.498448] pci 0000:85:00.0:   bridge window [mem 0xd3b00000-0xd3bfffff]
[    7.505545] pci 0000:87:00.0: [8086:0a54] type 00 class 0x010802
[    7.511844] pci 0000:87:00.0: reg 0x10: [mem 0xd3a00000-0xd3a03fff 64bit]
[    7.519102] pci 0000:85:01.0: PCI bridge to [bus 87]
[    7.524356] pci 0000:85:01.0:   bridge window [mem 0xd3a00000-0xd3afffff]
[    7.531460] pci 0000:88:00.0: [8086:0a54] type 00 class 0x010802
[    7.537764] pci 0000:88:00.0: reg 0x10: [mem 0xd3900000-0xd3903fff 64bit]
[    7.545020] pci 0000:85:02.0: PCI bridge to [bus 88]
[    7.550274] pci 0000:85:02.0:   bridge window [mem 0xd3900000-0xd39fffff]
[    7.557388] pci 0000:89:00.0: [8086:0a54] type 00 class 0x010802
[    7.563690] pci 0000:89:00.0: reg 0x10: [mem 0xd3800000-0xd3803fff 64bit]
[    7.570948] pci 0000:85:03.0: PCI bridge to [bus 89]
[    7.576198] pci 0000:85:03.0:   bridge window [mem 0xd3800000-0xd38fffff]
[    7.583282] pci_bus 0000:85: on NUMA node 1
[    7.583404] ACPI: PCI Root Bridge [PC08] (domain 0000 [bus ae-d6])
[    7.589858] acpi PNP0A08:08: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    7.599439] acpi PNP0A08:08: PCIe AER handled by firmware
[    7.610808] acpi PNP0A08:08: _OSC: platform does not support [SHPCHotplug LTR]
[    7.618631] acpi PNP0A08:08: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    7.626778] acpi PNP0A08:08: FADT indicates ASPM is unsupported, using BIOS configuration
[    7.635595] PCI host bridge to bus 0000:ae
[    7.640014] pci_bus 0000:ae: root bus resource [io  0xc000-0xdfff window]
[    7.647008] pci_bus 0000:ae: root bus resource [mem 0xe1000000-0xee7fffff window]
[    7.655008] pci_bus 0000:ae: root bus resource [mem 0x386000000000-0x386fffffffff window]
[    7.663637] pci_bus 0000:ae: root bus resource [bus ae-d6]
[    7.669407] pci 0000:ae:00.0: [8086:2030] type 01 class 0x060400
[    7.675717] pci 0000:ae:00.0: enabling Extended Tags
[    7.681041] pci 0000:ae:00.0: PME# supported from D0 D3hot D3cold
[    7.687533] pci 0000:ae:01.0: [8086:2031] type 01 class 0x060400
[    7.693845] pci 0000:ae:01.0: enabling Extended Tags
[    7.699173] pci 0000:ae:01.0: PME# supported from D0 D3hot D3cold
[    7.705611] pci 0000:ae:02.0: [8086:2032] type 01 class 0x060400
[    7.712017] pci 0000:ae:02.0: enabling Extended Tags
[    7.717253] pci 0000:ae:02.0: PME# supported from D0 D3hot D3cold
[    7.723699] pci 0000:ae:03.0: [8086:2033] type 01 class 0x060400
[    7.730009] pci 0000:ae:03.0: enabling Extended Tags
[    7.735338] pci 0000:ae:03.0: PME# supported from D0 D3hot D3cold
[    7.741785] pci 0000:ae:05.0: [8086:2034] type 00 class 0x088000
[    7.748182] pci 0000:ae:05.2: [8086:2035] type 00 class 0x088000
[    7.754575] pci 0000:ae:05.4: [8086:2036] type 00 class 0x080020
[    7.761008] pci 0000:ae:05.4: reg 0x10: [mem 0xe1800000-0xe1800fff]
[    7.767521] pci 0000:ae:08.0: [8086:2066] type 00 class 0x088000
[    7.773919] pci 0000:ae:09.0: [8086:2066] type 00 class 0x088000
[    7.780309] pci 0000:ae:0a.0: [8086:2040] type 00 class 0x088000
[    7.786691] pci 0000:ae:0a.1: [8086:2041] type 00 class 0x088000
[    7.793071] pci 0000:ae:0a.2: [8086:2042] type 00 class 0x088000
[    7.799453] pci 0000:ae:0a.3: [8086:2043] type 00 class 0x088000
[    7.805836] pci 0000:ae:0a.4: [8086:2044] type 00 class 0x088000
[    7.812222] pci 0000:ae:0a.5: [8086:2045] type 00 class 0x088000
[    7.818604] pci 0000:ae:0a.6: [8086:2046] type 00 class 0x088000
[    7.825008] pci 0000:ae:0a.7: [8086:2047] type 00 class 0x088000
[    7.831367] pci 0000:ae:0b.0: [8086:2048] type 00 class 0x088000
[    7.837761] pci 0000:ae:0b.1: [8086:2049] type 00 class 0x088000
[    7.844144] pci 0000:ae:0b.2: [8086:204a] type 00 class 0x088000
[    7.850531] pci 0000:ae:0b.3: [8086:204b] type 00 class 0x088000
[    7.856918] pci 0000:ae:0c.0: [8086:2040] type 00 class 0x088000
[    7.863308] pci 0000:ae:0c.1: [8086:2041] type 00 class 0x088000
[    7.869692] pci 0000:ae:0c.2: [8086:2042] type 00 class 0x088000
[    7.876076] pci 0000:ae:0c.3: [8086:2043] type 00 class 0x088000
[    7.882458] pci 0000:ae:0c.4: [8086:2044] type 00 class 0x088000
[    7.888844] pci 0000:ae:0c.5: [8086:2045] type 00 class 0x088000
[    7.895234] pci 0000:ae:0c.6: [8086:2046] type 00 class 0x088000
[    7.901627] pci 0000:ae:0c.7: [8086:2047] type 00 class 0x088000
[    7.908010] pci 0000:ae:0d.0: [8086:2048] type 00 class 0x088000
[    7.914377] pci 0000:ae:0d.1: [8086:2049] type 00 class 0x088000
[    7.920764] pci 0000:ae:0d.2: [8086:204a] type 00 class 0x088000
[    7.927152] pci 0000:ae:0d.3: [8086:204b] type 00 class 0x088000
[    7.933584] pci 0000:ae:00.0: PCI bridge to [bus af]
[    7.939008] pci 0000:ae:00.0:   bridge window [mem 0xe1600000-0xe17fffff]
[    7.946008] pci 0000:ae:01.0: PCI bridge to [bus b0]
[    7.951177] pci 0000:ae:01.0:   bridge window [mem 0xe1400000-0xe15fffff]
[    7.958276] pci 0000:ae:02.0: PCI bridge to [bus b1]
[    7.963521] pci 0000:ae:02.0:   bridge window [mem 0xe1200000-0xe13fffff]
[    7.970613] pci 0000:ae:03.0: PCI bridge to [bus b2]
[    7.975859] pci 0000:ae:03.0:   bridge window [mem 0xe1000000-0xe11fffff]
[    7.983008] pci_bus 0000:ae: on NUMA node 1
[    7.983067] ACPI: PCI Root Bridge [PC09] (domain 0000 [bus d7-ff])
[    7.989528] acpi PNP0A08:09: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    7.999112] acpi PNP0A08:09: PCIe AER handled by firmware
[    8.005008] acpi PNP0A08:09: _OSC: platform does not support [SHPCHotplug LTR]
[    8.012674] acpi PNP0A08:09: _OSC: OS now controls [PCIeHotplug PME PCIeCapability]
[    8.021008] acpi PNP0A08:09: FADT indicates ASPM is unsupported, using BIOS configuration
[    8.029652] PCI host bridge to bus 0000:d7
[    8.034043] pci_bus 0000:d7: root bus resource [io  0xe000-0xffff window]
[    8.041105] pci_bus 0000:d7: root bus resource [mem 0xee800000-0xfbffffff window]
[    8.049085] pci_bus 0000:d7: root bus resource [mem 0x387000000000-0x387fffffffff window]
[    8.057745] pci_bus 0000:d7: root bus resource [bus d7-ff]
[    8.063517] pci 0000:d7:05.0: [8086:2034] type 00 class 0x088000
[    8.070008] pci 0000:d7:05.2: [8086:2035] type 00 class 0x088000
[    8.076296] pci 0000:d7:05.4: [8086:2036] type 00 class 0x080020
[    8.082588] pci 0000:d7:05.4: reg 0x10: [mem 0xee800000-0xee800fff]
[    8.089243] pci 0000:d7:0e.0: [8086:2058] type 00 class 0x110100
[    8.095627] pci 0000:d7:0e.1: [8086:2059] type 00 class 0x088000
[    8.102014] pci 0000:d7:0f.0: [8086:2058] type 00 class 0x110100
[    8.108372] pci 0000:d7:0f.1: [8086:2059] type 00 class 0x088000
[    8.115008] pci 0000:d7:10.0: [8086:2058] type 00 class 0x110100
[    8.121142] pci 0000:d7:10.1: [8086:2059] type 00 class 0x088000
[    8.127523] pci 0000:d7:12.0: [8086:204c] type 00 class 0x110100
[    8.134007] pci 0000:d7:12.1: [8086:204d] type 00 class 0x110100
[    8.140257] pci 0000:d7:12.2: [8086:204e] type 00 class 0x088000
[    8.146626] pci 0000:d7:12.4: [8086:204c] type 00 class 0x110100
[    8.153022] pci 0000:d7:12.5: [8086:204d] type 00 class 0x110100
[    8.159357] pci 0000:d7:15.0: [8086:2018] type 00 class 0x088000
[    8.165725] pci 0000:d7:16.0: [8086:2018] type 00 class 0x088000
[    8.172091] pci 0000:d7:16.4: [8086:2018] type 00 class 0x088000
[    8.178455] pci 0000:d7:17.0: [8086:2018] type 00 class 0x088000
[    8.185008] pci_bus 0000:d7: on NUMA node 1
[    8.186061] iommu: Default domain type: Passthrough (set via kernel command line)
[    8.194016] pci 0000:03:00.0: vgaarb: setting as boot VGA device
[    8.200063] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    8.209184] pci 0000:03:00.0: vgaarb: bridge control possible
[    8.215210] vgaarb: loaded
[    8.218372] SCSI subsystem initialized
[    8.222435] ACPI: bus type USB registered
[    8.227008] usbcore: registered new interface driver usbfs
[    8.233008] usbcore: registered new interface driver hub
[    8.238303] usbcore: registered new device driver usb
[    8.244007] pps_core: LinuxPPS API ver. 1 registered
[    8.249014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    8.258500] PTP clock support registered
[    8.263093] EDAC MC: Ver: 3.0.0
[    8.267044] PCI: Using ACPI for IRQ routing
[    8.274829] PCI: pci_cache_line_size set to 64 bytes
[    8.275182] Expanded resource Reserved due to conflict with PNP0003:00
[    8.281988] e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff]
[    8.281989] e820: reserve RAM buffer [mem 0x557b9000-0x57ffffff]
[    8.281990] e820: reserve RAM buffer [mem 0x60fd5000-0x63ffffff]
[    8.281991] e820: reserve RAM buffer [mem 0x6c5cf000-0x6fffffff]
[    8.281993] e820: reserve RAM buffer [mem 0x6f800000-0x6fffffff]
[    8.283011] NetLabel: Initializing
[    8.287008] NetLabel:  domain hash size = 128
[    8.291159] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    8.297108] NetLabel:  unlabeled traffic allowed by default
[    8.303972] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    8.310424] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[    8.319155] clocksource: Switched to clocksource tsc-early
[    8.344854] VFS: Disk quotas dquot_6.6.0
[    8.349177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    8.356465] pnp: PnP ACPI init
[    8.360434] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[    8.360684] system 00:01: [io  0x0500-0x053f] has been reserved
[    8.366897] system 00:01: [io  0x0400-0x047f] has been reserved
[    8.373123] system 00:01: [io  0x0540-0x057f] has been reserved
[    8.379337] system 00:01: [io  0x0600-0x061f] has been reserved
[    8.385543] system 00:01: [io  0x0ca0-0x0ca5] has been reserved
[    8.391753] system 00:01: [io  0x0880-0x0883] has been reserved
[    8.397959] system 00:01: [io  0x0800-0x081f] has been reserved
[    8.404173] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved
[    8.411076] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved
[    8.417977] system 00:01: [mem 0xff000000-0xffffffff] has been reserved
[    8.424876] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved
[    8.431775] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved
[    8.438678] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved
[    8.445585] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved
[    8.452496] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    8.452705] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active)
[    8.452876] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active)
[    8.453117] system 00:04: [mem 0xfd000000-0xfdabffff] has been reserved
[    8.460027] system 00:04: [mem 0xfdad0000-0xfdadffff] has been reserved
[    8.466932] system 00:04: [mem 0xfdb00000-0xfdffffff] has been reserved
[    8.473832] system 00:04: [mem 0xfe000000-0xfe00ffff] has been reserved
[    8.480737] system 00:04: [mem 0xfe011000-0xfe01ffff] has been reserved
[    8.487639] system 00:04: [mem 0xfe036000-0xfe03bfff] has been reserved
[    8.494538] system 00:04: [mem 0xfe03d000-0xfe3fffff] has been reserved
[    8.501437] system 00:04: [mem 0xfe410000-0xfe7fffff] has been reserved
[    8.508343] system 00:04: Plug and Play ACPI device, IDs PNP0c02 (active)
[    8.508713] system 00:05: [io  0x1000-0x10fe] has been reserved
[    8.514926] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active)
[    8.515594] pnp: PnP ACPI: found 6 devices
[    8.525849] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    8.535257] pci 0000:18:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[    8.545671] pci 0000:19:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[    8.556091] pci 0000:19:00.1: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[    8.566514] pci 0000:3b:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[    8.576930] pci 0000:3b:00.1: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[    8.587356] pci 0000:00:1c.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    8.596047] pci 0000:00:1c.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    8.608032] pci 0000:00:1c.0: BAR 15: assigned [mem 0x380000000000-0x3800001fffff 64bit pref]
[    8.617068] pci 0000:00:1c.0: BAR 13: assigned [io  0x3000-0x3fff]
[    8.623537] pci 0000:00:1c.0: PCI bridge to [bus 01]
[    8.628793] pci 0000:00:1c.0:   bridge window [io  0x3000-0x3fff]
[    8.635179] pci 0000:00:1c.0:   bridge window [mem 0x92a00000-0x92bfffff]
[    8.642261] pci 0000:00:1c.0:   bridge window [mem 0x380000000000-0x3800001fffff 64bit pref]
[    8.651213] pci 0000:02:00.0: PCI bridge to [bus 03]
[    8.656473] pci 0000:02:00.0:   bridge window [mem 0x92000000-0x928fffff]
[    8.663553] pci 0000:02:00.0:   bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[    8.677443] pci 0000:00:1c.4: PCI bridge to [bus 02-03]
[    8.682963] pci 0000:00:1c.4:   bridge window [mem 0x92000000-0x928fffff]
[    8.690038] pci 0000:00:1c.4:   bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[    8.698294] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    8.704770] pci_bus 0000:00: resource 5 [io  0x1000-0x3fff window]
[    8.711244] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    8.718406] pci_bus 0000:00: resource 7 [mem 0xfe010000-0xfe010fff window]
[    8.725571] pci_bus 0000:00: resource 8 [mem 0x90000000-0x9d7fffff window]
[    8.732730] pci_bus 0000:00: resource 9 [mem 0x380000000000-0x380fffffffff window]
[    8.740808] pci_bus 0000:01: resource 0 [io  0x3000-0x3fff]
[    8.746673] pci_bus 0000:01: resource 1 [mem 0x92a00000-0x92bfffff]
[    8.753234] pci_bus 0000:01: resource 2 [mem 0x380000000000-0x3800001fffff 64bit pref]
[    8.761659] pci_bus 0000:02: resource 1 [mem 0x92000000-0x928fffff]
[    8.768213] pci_bus 0000:02: resource 2 [mem 0x91000000-0x91ffffff 64bit pref]
[    8.775941] pci_bus 0000:03: resource 1 [mem 0x92000000-0x928fffff]
[    8.782497] pci_bus 0000:03: resource 2 [mem 0x91000000-0x91ffffff 64bit pref]
[    8.790329] pci 0000:17:02.0: BAR 14: assigned [mem 0x9d800000-0x9d9fffff]
[    8.797490] pci 0000:18:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[    8.804829] pci 0000:18:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[    8.812732] pci 0000:17:00.0: PCI bridge to [bus 18]
[    8.817987] pci 0000:17:00.0:   bridge window [io  0x4000-0x4fff]
[    8.824371] pci 0000:17:00.0:   bridge window [mem 0xa2a00000-0xa2bfffff]
[    8.831454] pci 0000:19:00.0: BAR 6: assigned [mem 0x9d800000-0x9d8fffff pref]
[    8.839187] pci 0000:19:00.1: BAR 6: assigned [mem 0x9d900000-0x9d9fffff pref]
[    8.846919] pci 0000:17:02.0: PCI bridge to [bus 19-1a]
[    8.852440] pci 0000:17:02.0:   bridge window [mem 0x9d800000-0x9d9fffff]
[    8.859517] pci 0000:17:02.0:   bridge window [mem 0x9e000000-0xa29fffff 64bit pref]
[    8.867767] pci_bus 0000:17: resource 4 [io  0x4000-0x5fff window]
[    8.874245] pci_bus 0000:17: resource 5 [mem 0x9d800000-0xaaffffff window]
[    8.881418] pci_bus 0000:17: resource 6 [mem 0x381000000000-0x381fffffffff window]
[    8.889500] pci_bus 0000:18: resource 0 [io  0x4000-0x4fff]
[    8.895364] pci_bus 0000:18: resource 1 [mem 0xa2a00000-0xa2bfffff]
[    8.901934] pci_bus 0000:19: resource 1 [mem 0x9d800000-0x9d9fffff]
[    8.908488] pci_bus 0000:19: resource 2 [mem 0x9e000000-0xa29fffff 64bit pref]
[    8.916240] pci 0000:3a:00.0: BAR 14: assigned [mem 0xab000000-0xab1fffff]
[    8.923405] pci 0000:3b:00.0: BAR 6: assigned [mem 0xab000000-0xab0fffff pref]
[    8.931143] pci 0000:3b:00.1: BAR 6: assigned [mem 0xab100000-0xab1fffff pref]
[    8.938879] pci 0000:3a:00.0: PCI bridge to [bus 3b]
[    8.944138] pci 0000:3a:00.0:   bridge window [mem 0xab000000-0xab1fffff]
[    8.951224] pci 0000:3a:00.0:   bridge window [mem 0xac000000-0xafffffff 64bit pref]
[    8.959477] pci_bus 0000:3a: resource 4 [io  0x6000-0x7fff window]
[    8.965951] pci_bus 0000:3a: resource 5 [mem 0xab000000-0xb87fffff window]
[    8.973113] pci_bus 0000:3a: resource 6 [mem 0x382000000000-0x382fffffffff window]
[    8.981196] pci_bus 0000:3b: resource 1 [mem 0xab000000-0xab1fffff]
[    8.987752] pci_bus 0000:3b: resource 2 [mem 0xac000000-0xafffffff 64bit pref]
[    8.995496] pci_bus 0000:5d: resource 4 [io  0x8000-0x9fff window]
[    9.001972] pci_bus 0000:5d: resource 5 [mem 0xb8800000-0xc5ffffff window]
[    9.009139] pci_bus 0000:5d: resource 6 [mem 0x383000000000-0x383fffffffff window]
[    9.017241] pci_bus 0000:80: resource 4 [mem 0xc6000000-0xd37fffff window]
[    9.024407] pci_bus 0000:80: resource 5 [mem 0x384000000000-0x384fffffffff window]
[    9.032490] pci 0000:85:00.0: bridge window [io  0x1000-0x0fff] to [bus 86] add_size 1000
[    9.041176] pci 0000:85:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 86] add_size 200000 add_align 100000
[    9.053152] pci 0000:85:01.0: bridge window [io  0x1000-0x0fff] to [bus 87] add_size 1000
[    9.061837] pci 0000:85:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 87] add_size 200000 add_align 100000
[    9.073816] pci 0000:85:02.0: bridge window [io  0x1000-0x0fff] to [bus 88] add_size 1000
[    9.082502] pci 0000:85:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 88] add_size 200000 add_align 100000
[    9.094478] pci 0000:85:03.0: bridge window [io  0x1000-0x0fff] to [bus 89] add_size 1000
[    9.103165] pci 0000:85:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 89] add_size 200000 add_align 100000
[    9.115154] pci 0000:85:00.0: BAR 15: assigned [mem 0x385000000000-0x3850001fffff 64bit pref]
[    9.124185] pci 0000:85:01.0: BAR 15: assigned [mem 0x385000200000-0x3850003fffff 64bit pref]
[    9.133221] pci 0000:85:02.0: BAR 15: assigned [mem 0x385000400000-0x3850005fffff 64bit pref]
[    9.142258] pci 0000:85:03.0: BAR 15: assigned [mem 0x385000600000-0x3850007fffff 64bit pref]
[    9.151299] pci 0000:85:00.0: BAR 13: assigned [io  0xa000-0xafff]
[    9.157773] pci 0000:85:01.0: BAR 13: assigned [io  0xb000-0xbfff]
[    9.164241] pci 0000:85:02.0: BAR 13: no space for [io  size 0x1000]
[    9.170888] pci 0000:85:02.0: BAR 13: failed to assign [io  size 0x1000]
[    9.177876] pci 0000:85:03.0: BAR 13: no space for [io  size 0x1000]
[    9.184514] pci 0000:85:03.0: BAR 13: failed to assign [io  size 0x1000]
[    9.191502] pci 0000:85:03.0: BAR 13: assigned [io  0xa000-0xafff]
[    9.197966] pci 0000:85:02.0: BAR 13: assigned [io  0xb000-0xbfff]
[    9.204436] pci 0000:85:01.0: BAR 13: no space for [io  size 0x1000]
[    9.211077] pci 0000:85:01.0: BAR 13: failed to assign [io  size 0x1000]
[    9.218075] pci 0000:85:00.0: BAR 13: no space for [io  size 0x1000]
[    9.224719] pci 0000:85:00.0: BAR 13: failed to assign [io  size 0x1000]
[    9.231706] pci 0000:85:00.0: PCI bridge to [bus 86]
[    9.236968] pci 0000:85:00.0:   bridge window [mem 0xd3b00000-0xd3bfffff]
[    9.244048] pci 0000:85:00.0:   bridge window [mem 0x385000000000-0x3850001fffff 64bit pref]
[    9.252992] pci 0000:85:01.0: PCI bridge to [bus 87]
[    9.258250] pci 0000:85:01.0:   bridge window [mem 0xd3a00000-0xd3afffff]
[    9.265346] pci 0000:85:01.0:   bridge window [mem 0x385000200000-0x3850003fffff 64bit pref]
[    9.274292] pci 0000:85:02.0: PCI bridge to [bus 88]
[    9.279561] pci 0000:85:02.0:   bridge window [io  0xb000-0xbfff]
[    9.285942] pci 0000:85:02.0:   bridge window [mem 0xd3900000-0xd39fffff]
[    9.293015] pci 0000:85:02.0:   bridge window [mem 0x385000400000-0x3850005fffff 64bit pref]
[    9.301955] pci 0000:85:03.0: PCI bridge to [bus 89]
[    9.307214] pci 0000:85:03.0:   bridge window [io  0xa000-0xafff]
[    9.313602] pci 0000:85:03.0:   bridge window [mem 0xd3800000-0xd38fffff]
[    9.320676] pci 0000:85:03.0:   bridge window [mem 0x385000600000-0x3850007fffff 64bit pref]
[    9.329622] pci_bus 0000:85: resource 4 [io  0xa000-0xbfff window]
[    9.336092] pci_bus 0000:85: resource 5 [mem 0xd3800000-0xe0ffffff window]
[    9.343262] pci_bus 0000:85: resource 6 [mem 0x385000000000-0x385fffffffff window]
[    9.351352] pci_bus 0000:86: resource 1 [mem 0xd3b00000-0xd3bfffff]
[    9.357913] pci_bus 0000:86: resource 2 [mem 0x385000000000-0x3850001fffff 64bit pref]
[    9.366332] pci_bus 0000:87: resource 1 [mem 0xd3a00000-0xd3afffff]
[    9.372889] pci_bus 0000:87: resource 2 [mem 0x385000200000-0x3850003fffff 64bit pref]
[    9.381320] pci_bus 0000:88: resource 0 [io  0xb000-0xbfff]
[    9.387191] pci_bus 0000:88: resource 1 [mem 0xd3900000-0xd39fffff]
[    9.393751] pci_bus 0000:88: resource 2 [mem 0x385000400000-0x3850005fffff 64bit pref]
[    9.402174] pci_bus 0000:89: resource 0 [io  0xa000-0xafff]
[    9.408046] pci_bus 0000:89: resource 1 [mem 0xd3800000-0xd38fffff]
[    9.414605] pci_bus 0000:89: resource 2 [mem 0x385000600000-0x3850007fffff 64bit pref]
[    9.423044] pci 0000:ae:00.0: bridge window [io  0x1000-0x0fff] to [bus af] add_size 1000
[    9.431734] pci 0000:ae:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus af] add_size 200000 add_align 100000
[    9.443714] pci 0000:ae:01.0: bridge window [io  0x1000-0x0fff] to [bus b0] add_size 1000
[    9.452400] pci 0000:ae:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus b0] add_size 200000 add_align 100000
[    9.464378] pci 0000:ae:02.0: bridge window [io  0x1000-0x0fff] to [bus b1] add_size 1000
[    9.473064] pci 0000:ae:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus b1] add_size 200000 add_align 100000
[    9.485041] pci 0000:ae:03.0: bridge window [io  0x1000-0x0fff] to [bus b2] add_size 1000
[    9.493728] pci 0000:ae:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus b2] add_size 200000 add_align 100000
[    9.505716] pci 0000:ae:00.0: BAR 15: assigned [mem 0x386000000000-0x3860001fffff 64bit pref]
[    9.514742] pci 0000:ae:01.0: BAR 15: assigned [mem 0x386000200000-0x3860003fffff 64bit pref]
[    9.523771] pci 0000:ae:02.0: BAR 15: assigned [mem 0x386000400000-0x3860005fffff 64bit pref]
[    9.532800] pci 0000:ae:03.0: BAR 15: assigned [mem 0x386000600000-0x3860007fffff 64bit pref]
[    9.541832] pci 0000:ae:00.0: BAR 13: assigned [io  0xc000-0xcfff]
[    9.548302] pci 0000:ae:01.0: BAR 13: assigned [io  0xd000-0xdfff]
[    9.554775] pci 0000:ae:02.0: BAR 13: no space for [io  size 0x1000]
[    9.561418] pci 0000:ae:02.0: BAR 13: failed to assign [io  size 0x1000]
[    9.568411] pci 0000:ae:03.0: BAR 13: no space for [io  size 0x1000]
[    9.575052] pci 0000:ae:03.0: BAR 13: failed to assign [io  size 0x1000]
[    9.582041] pci 0000:ae:03.0: BAR 13: assigned [io  0xc000-0xcfff]
[    9.588511] pci 0000:ae:02.0: BAR 13: assigned [io  0xd000-0xdfff]
[    9.594979] pci 0000:ae:01.0: BAR 13: no space for [io  size 0x1000]
[    9.601622] pci 0000:ae:01.0: BAR 13: failed to assign [io  size 0x1000]
[    9.608628] pci 0000:ae:00.0: BAR 13: no space for [io  size 0x1000]
[    9.615273] pci 0000:ae:00.0: BAR 13: failed to assign [io  size 0x1000]
[    9.622261] pci 0000:ae:00.0: PCI bridge to [bus af]
[    9.627523] pci 0000:ae:00.0:   bridge window [mem 0xe1600000-0xe17fffff]
[    9.634599] pci 0000:ae:00.0:   bridge window [mem 0x386000000000-0x3860001fffff 64bit pref]
[    9.649179] pci 0000:ae:01.0: PCI bridge to [bus b0]
[    9.654440] pci 0000:ae:01.0:   bridge window [mem 0xe1400000-0xe15fffff]
[    9.661515] pci 0000:ae:01.0:   bridge window [mem 0x386000200000-0x3860003fffff 64bit pref]
[    9.670462] pci 0000:ae:02.0: PCI bridge to [bus b1]
[    9.675725] pci 0000:ae:02.0:   bridge window [io  0xd000-0xdfff]
[    9.682111] pci 0000:ae:02.0:   bridge window [mem 0xe1200000-0xe13fffff]
[    9.689196] pci 0000:ae:02.0:   bridge window [mem 0x386000400000-0x3860005fffff 64bit pref]
[    9.698141] pci 0000:ae:03.0: PCI bridge to [bus b2]
[    9.703400] pci 0000:ae:03.0:   bridge window [io  0xc000-0xcfff]
[    9.709783] pci 0000:ae:03.0:   bridge window [mem 0xe1000000-0xe11fffff]
[    9.716858] pci 0000:ae:03.0:   bridge window [mem 0x386000600000-0x3860007fffff 64bit pref]
[    9.725810] pci_bus 0000:ae: resource 4 [io  0xc000-0xdfff window]
[    9.732280] pci_bus 0000:ae: resource 5 [mem 0xe1000000-0xee7fffff window]
[    9.739443] pci_bus 0000:ae: resource 6 [mem 0x386000000000-0x386fffffffff window]
[    9.747524] pci_bus 0000:af: resource 1 [mem 0xe1600000-0xe17fffff]
[    9.754078] pci_bus 0000:af: resource 2 [mem 0x386000000000-0x3860001fffff 64bit pref]
[    9.762503] pci_bus 0000:b0: resource 1 [mem 0xe1400000-0xe15fffff]
[    9.769063] pci_bus 0000:b0: resource 2 [mem 0x386000200000-0x3860003fffff 64bit pref]
[    9.777492] pci_bus 0000:b1: resource 0 [io  0xd000-0xdfff]
[    9.783354] pci_bus 0000:b1: resource 1 [mem 0xe1200000-0xe13fffff]
[    9.789906] pci_bus 0000:b1: resource 2 [mem 0x386000400000-0x3860005fffff 64bit pref]
[    9.798330] pci_bus 0000:b2: resource 0 [io  0xc000-0xcfff]
[    9.804200] pci_bus 0000:b2: resource 1 [mem 0xe1000000-0xe11fffff]
[    9.810761] pci_bus 0000:b2: resource 2 [mem 0x386000600000-0x3860007fffff 64bit pref]
[    9.819219] pci_bus 0000:d7: resource 4 [io  0xe000-0xffff window]
[    9.825694] pci_bus 0000:d7: resource 5 [mem 0xee800000-0xfbffffff window]
[    9.832854] pci_bus 0000:d7: resource 6 [mem 0x387000000000-0x387fffffffff window]
[    9.841092] NET: Registered protocol family 2
[    9.846525] tcp_listen_portaddr_hash hash table entries: 65536 (order: 10, 4194304 bytes)
[    9.856154] TCP established hash table entries: 524288 (order: 10, 4194304 bytes)
[    9.864942] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
[    9.872562] TCP: Hash tables configured (established 524288 bind 65536)
[    9.880191] UDP hash table entries: 65536 (order: 11, 8388608 bytes)
[    9.888769] UDP-Lite hash table entries: 65536 (order: 11, 8388608 bytes)
[    9.897661] NET: Registered protocol family 1
[    9.902323] NET: Registered protocol family 44
[    9.908342] pci 0000:03:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    9.918368] PCI: CLS 0 bytes, default 64
[    9.922625] Unpacking initramfs...
[   10.948921] Freeing initrd memory: 73096K
[   10.953370] DMAR: [Firmware Bug]: RMRR entry for device 18:00.0 is broken - applying workaround
[   10.963662] DMAR: dmar5: Using Queued invalidation
[   10.968757] DMAR: dmar4: Using Queued invalidation
[   10.973847] DMAR: dmar2: Using Queued invalidation
[   10.978941] DMAR: dmar1: Using Queued invalidation
[   10.984034] DMAR: dmar7: Using Queued invalidation
[   10.990333] pci 0000:00:00.0: Adding to iommu group 0
[   10.995953] pci 0000:00:05.0: Adding to iommu group 1
[   11.001368] pci 0000:00:05.2: Adding to iommu group 1
[   11.006786] pci 0000:00:05.4: Adding to iommu group 1
[   11.012386] pci 0000:00:08.0: Adding to iommu group 2
[   11.017797] pci 0000:00:08.1: Adding to iommu group 2
[   11.023215] pci 0000:00:08.2: Adding to iommu group 2
[   11.028754] pci 0000:00:11.0: Adding to iommu group 3
[   11.034169] pci 0000:00:11.5: Adding to iommu group 3
[   11.039700] pci 0000:00:14.0: Adding to iommu group 4
[   11.045115] pci 0000:00:14.2: Adding to iommu group 4
[   11.050705] pci 0000:00:16.0: Adding to iommu group 5
[   11.056124] pci 0000:00:16.1: Adding to iommu group 5
[   11.061539] pci 0000:00:16.4: Adding to iommu group 5
[   11.066952] pci 0000:00:17.0: Adding to iommu group 6
[   11.072432] pci 0000:00:1c.0: Adding to iommu group 7
[   11.077849] pci 0000:00:1c.4: Adding to iommu group 8
[   11.083495] pci 0000:00:1f.0: Adding to iommu group 9
[   11.088905] pci 0000:00:1f.2: Adding to iommu group 9
[   11.094320] pci 0000:00:1f.4: Adding to iommu group 9
[   11.099731] pci 0000:00:1f.5: Adding to iommu group 9
[   11.105147] pci 0000:02:00.0: Adding to iommu group 10
[   11.111759] pci 0000:02:00.0: Using iommu dma mapping
[   11.117112] pci 0000:03:00.0: Adding to iommu group 10
[   11.122623] pci 0000:17:00.0: Adding to iommu group 11
[   11.128129] pci 0000:17:02.0: Adding to iommu group 12
[   11.133813] pci 0000:17:05.0: Adding to iommu group 13
[   11.139335] pci 0000:17:05.2: Adding to iommu group 13
[   11.144838] pci 0000:17:05.4: Adding to iommu group 13
[   11.150806] pci 0000:17:08.0: Adding to iommu group 14
[   11.156330] pci 0000:17:08.1: Adding to iommu group 14
[   11.161832] pci 0000:17:08.2: Adding to iommu group 14
[   11.167337] pci 0000:17:08.3: Adding to iommu group 14
[   11.172837] pci 0000:17:08.4: Adding to iommu group 14
[   11.178343] pci 0000:17:08.5: Adding to iommu group 14
[   11.183841] pci 0000:17:08.6: Adding to iommu group 14
[   11.189341] pci 0000:17:08.7: Adding to iommu group 14
[   11.195310] pci 0000:17:09.0: Adding to iommu group 15
[   11.200816] pci 0000:17:09.1: Adding to iommu group 15
[   11.206324] pci 0000:17:09.2: Adding to iommu group 15
[   11.211828] pci 0000:17:09.3: Adding to iommu group 15
[   11.217335] pci 0000:17:09.4: Adding to iommu group 15
[   11.222836] pci 0000:17:09.5: Adding to iommu group 15
[   11.228339] pci 0000:17:09.6: Adding to iommu group 15
[   11.233845] pci 0000:17:09.7: Adding to iommu group 15
[   11.239820] pci 0000:17:0a.0: Adding to iommu group 16
[   11.245334] pci 0000:17:0a.1: Adding to iommu group 16
[   11.250843] pci 0000:17:0a.2: Adding to iommu group 16
[   11.256352] pci 0000:17:0a.3: Adding to iommu group 16
[   11.261857] pci 0000:17:0a.4: Adding to iommu group 16
[   11.267367] pci 0000:17:0a.5: Adding to iommu group 16
[   11.272876] pci 0000:17:0a.6: Adding to iommu group 16
[   11.278382] pci 0000:17:0a.7: Adding to iommu group 16
[   11.284119] pci 0000:17:0b.0: Adding to iommu group 17
[   11.289622] pci 0000:17:0b.1: Adding to iommu group 17
[   11.295131] pci 0000:17:0b.2: Adding to iommu group 17
[   11.300640] pci 0000:17:0b.3: Adding to iommu group 17
[   11.306608] pci 0000:17:0e.0: Adding to iommu group 18
[   11.312173] pci 0000:17:0e.1: Adding to iommu group 18
[   11.317681] pci 0000:17:0e.2: Adding to iommu group 18
[   11.323192] pci 0000:17:0e.3: Adding to iommu group 18
[   11.328696] pci 0000:17:0e.4: Adding to iommu group 18
[   11.334202] pci 0000:17:0e.5: Adding to iommu group 18
[   11.339715] pci 0000:17:0e.6: Adding to iommu group 18
[   11.345232] pci 0000:17:0e.7: Adding to iommu group 18
[   11.351211] pci 0000:17:0f.0: Adding to iommu group 19
[   11.356724] pci 0000:17:0f.1: Adding to iommu group 19
[   11.362237] pci 0000:17:0f.2: Adding to iommu group 19
[   11.367744] pci 0000:17:0f.3: Adding to iommu group 19
[   11.373252] pci 0000:17:0f.4: Adding to iommu group 19
[   11.378758] pci 0000:17:0f.5: Adding to iommu group 19
[   11.384267] pci 0000:17:0f.6: Adding to iommu group 19
[   11.389773] pci 0000:17:0f.7: Adding to iommu group 19
[   11.395740] pci 0000:17:10.0: Adding to iommu group 20
[   11.401255] pci 0000:17:10.1: Adding to iommu group 20
[   11.406763] pci 0000:17:10.2: Adding to iommu group 20
[   11.412273] pci 0000:17:10.3: Adding to iommu group 20
[   11.417803] pci 0000:17:10.4: Adding to iommu group 20
[   11.423316] pci 0000:17:10.5: Adding to iommu group 20
[   11.428827] pci 0000:17:10.6: Adding to iommu group 20
[   11.434354] pci 0000:17:10.7: Adding to iommu group 20
[   11.440087] pci 0000:17:11.0: Adding to iommu group 21
[   11.445597] pci 0000:17:11.1: Adding to iommu group 21
[   11.451113] pci 0000:17:11.2: Adding to iommu group 21
[   11.456628] pci 0000:17:11.3: Adding to iommu group 21
[   11.462363] pci 0000:17:1d.0: Adding to iommu group 22
[   11.467881] pci 0000:17:1d.1: Adding to iommu group 22
[   11.473395] pci 0000:17:1d.2: Adding to iommu group 22
[   11.478910] pci 0000:17:1d.3: Adding to iommu group 22
[   11.484824] pci 0000:17:1e.0: Adding to iommu group 23
[   11.490349] pci 0000:17:1e.1: Adding to iommu group 23
[   11.495866] pci 0000:17:1e.2: Adding to iommu group 23
[   11.501385] pci 0000:17:1e.3: Adding to iommu group 23
[   11.506896] pci 0000:17:1e.4: Adding to iommu group 23
[   11.512415] pci 0000:17:1e.5: Adding to iommu group 23
[   11.517927] pci 0000:17:1e.6: Adding to iommu group 23
[   11.523428] pci 0000:18:00.0: Adding to iommu group 24
[   11.529049] pci 0000:19:00.0: Adding to iommu group 25
[   11.534639] pci 0000:19:00.1: Adding to iommu group 26
[   11.540144] pci 0000:3a:00.0: Adding to iommu group 27
[   11.545837] pci 0000:3a:05.0: Adding to iommu group 28
[   11.551356] pci 0000:3a:05.2: Adding to iommu group 28
[   11.556884] pci 0000:3a:05.4: Adding to iommu group 28
[   11.562444] pci 0000:3a:08.0: Adding to iommu group 29
[   11.568013] pci 0000:3a:09.0: Adding to iommu group 30
[   11.573983] pci 0000:3a:0a.0: Adding to iommu group 31
[   11.579511] pci 0000:3a:0a.1: Adding to iommu group 31
[   11.585037] pci 0000:3a:0a.2: Adding to iommu group 31
[   11.590572] pci 0000:3a:0a.3: Adding to iommu group 31
[   11.596091] pci 0000:3a:0a.4: Adding to iommu group 31
[   11.601614] pci 0000:3a:0a.5: Adding to iommu group 31
[   11.607135] pci 0000:3a:0a.6: Adding to iommu group 31
[   11.612651] pci 0000:3a:0a.7: Adding to iommu group 31
[   11.618386] pci 0000:3a:0b.0: Adding to iommu group 32
[   11.623905] pci 0000:3a:0b.1: Adding to iommu group 32
[   11.629429] pci 0000:3a:0b.2: Adding to iommu group 32
[   11.634956] pci 0000:3a:0b.3: Adding to iommu group 32
[   11.640924] pci 0000:3a:0c.0: Adding to iommu group 33
[   11.646448] pci 0000:3a:0c.1: Adding to iommu group 33
[   11.651969] pci 0000:3a:0c.2: Adding to iommu group 33
[   11.657493] pci 0000:3a:0c.3: Adding to iommu group 33
[   11.668693] pci 0000:3a:0c.4: Adding to iommu group 33
[   11.674217] pci 0000:3a:0c.5: Adding to iommu group 33
[   11.679740] pci 0000:3a:0c.6: Adding to iommu group 33
[   11.685258] pci 0000:3a:0c.7: Adding to iommu group 33
[   11.690988] pci 0000:3a:0d.0: Adding to iommu group 34
[   11.696509] pci 0000:3a:0d.1: Adding to iommu group 34
[   11.702034] pci 0000:3a:0d.2: Adding to iommu group 34
[   11.707559] pci 0000:3a:0d.3: Adding to iommu group 34
[   11.713143] pci 0000:3b:00.0: Adding to iommu group 35
[   11.718708] pci 0000:3b:00.1: Adding to iommu group 36
[   11.724384] pci 0000:5d:05.0: Adding to iommu group 37
[   11.729912] pci 0000:5d:05.2: Adding to iommu group 37
[   11.735435] pci 0000:5d:05.4: Adding to iommu group 37
[   11.741055] pci 0000:5d:0e.0: Adding to iommu group 38
[   11.746585] pci 0000:5d:0e.1: Adding to iommu group 38
[   11.752203] pci 0000:5d:0f.0: Adding to iommu group 39
[   11.757732] pci 0000:5d:0f.1: Adding to iommu group 39
[   11.763347] pci 0000:5d:10.0: Adding to iommu group 40
[   11.768878] pci 0000:5d:10.1: Adding to iommu group 40
[   11.774674] pci 0000:5d:12.0: Adding to iommu group 41
[   11.780200] pci 0000:5d:12.1: Adding to iommu group 41
[   11.785729] pci 0000:5d:12.2: Adding to iommu group 41
[   11.791255] pci 0000:5d:12.4: Adding to iommu group 41
[   11.796778] pci 0000:5d:12.5: Adding to iommu group 41
[   11.802339] pci 0000:5d:15.0: Adding to iommu group 42
[   11.807956] pci 0000:5d:16.0: Adding to iommu group 43
[   11.813487] pci 0000:5d:16.4: Adding to iommu group 43
[   11.819045] pci 0000:5d:17.0: Adding to iommu group 44
[   11.824721] pci 0000:80:05.0: Adding to iommu group 45
[   11.830256] pci 0000:80:05.2: Adding to iommu group 45
[   11.835781] pci 0000:80:05.4: Adding to iommu group 45
[   11.841452] pci 0000:80:08.0: Adding to iommu group 46
[   11.846984] pci 0000:80:08.1: Adding to iommu group 46
[   11.852513] pci 0000:80:08.2: Adding to iommu group 46
[   11.858015] pci 0000:85:00.0: Adding to iommu group 47
[   11.863524] pci 0000:85:01.0: Adding to iommu group 48
[   11.869036] pci 0000:85:02.0: Adding to iommu group 49
[   11.874536] pci 0000:85:03.0: Adding to iommu group 50
[   11.880212] pci 0000:85:05.0: Adding to iommu group 51
[   11.885746] pci 0000:85:05.2: Adding to iommu group 51
[   11.891280] pci 0000:85:05.4: Adding to iommu group 51
[   11.897255] pci 0000:85:08.0: Adding to iommu group 52
[   11.902789] pci 0000:85:08.1: Adding to iommu group 52
[   11.908322] pci 0000:85:08.2: Adding to iommu group 52
[   11.913861] pci 0000:85:08.3: Adding to iommu group 52
[   11.919396] pci 0000:85:08.4: Adding to iommu group 52
[   11.924925] pci 0000:85:08.5: Adding to iommu group 52
[   11.930452] pci 0000:85:08.6: Adding to iommu group 52
[   11.935993] pci 0000:85:08.7: Adding to iommu group 52
[   11.941965] pci 0000:85:09.0: Adding to iommu group 53
[   11.947498] pci 0000:85:09.1: Adding to iommu group 53
[   11.953033] pci 0000:85:09.2: Adding to iommu group 53
[   11.958559] pci 0000:85:09.3: Adding to iommu group 53
[   11.964097] pci 0000:85:09.4: Adding to iommu group 53
[   11.969628] pci 0000:85:09.5: Adding to iommu group 53
[   11.975164] pci 0000:85:09.6: Adding to iommu group 53
[   11.980695] pci 0000:85:09.7: Adding to iommu group 53
[   11.986671] pci 0000:85:0a.0: Adding to iommu group 54
[   11.992212] pci 0000:85:0a.1: Adding to iommu group 54
[   11.997757] pci 0000:85:0a.2: Adding to iommu group 54
[   12.003293] pci 0000:85:0a.3: Adding to iommu group 54
[   12.008824] pci 0000:85:0a.4: Adding to iommu group 54
[   12.014360] pci 0000:85:0a.5: Adding to iommu group 54
[   12.019890] pci 0000:85:0a.6: Adding to iommu group 54
[   12.025427] pci 0000:85:0a.7: Adding to iommu group 54
[   12.031161] pci 0000:85:0b.0: Adding to iommu group 55
[   12.036700] pci 0000:85:0b.1: Adding to iommu group 55
[   12.042240] pci 0000:85:0b.2: Adding to iommu group 55
[   12.047778] pci 0000:85:0b.3: Adding to iommu group 55
[   12.053743] pci 0000:85:0e.0: Adding to iommu group 56
[   12.059285] pci 0000:85:0e.1: Adding to iommu group 56
[   12.064818] pci 0000:85:0e.2: Adding to iommu group 56
[   12.070354] pci 0000:85:0e.3: Adding to iommu group 56
[   12.075886] pci 0000:85:0e.4: Adding to iommu group 56
[   12.081420] pci 0000:85:0e.5: Adding to iommu group 56
[   12.086952] pci 0000:85:0e.6: Adding to iommu group 56
[   12.092487] pci 0000:85:0e.7: Adding to iommu group 56
[   12.098452] pci 0000:85:0f.0: Adding to iommu group 57
[   12.103995] pci 0000:85:0f.1: Adding to iommu group 57
[   12.109531] pci 0000:85:0f.2: Adding to iommu group 57
[   12.115074] pci 0000:85:0f.3: Adding to iommu group 57
[   12.120614] pci 0000:85:0f.4: Adding to iommu group 57
[   12.126158] pci 0000:85:0f.5: Adding to iommu group 57
[   12.131694] pci 0000:85:0f.6: Adding to iommu group 57
[   12.137233] pci 0000:85:0f.7: Adding to iommu group 57
[   12.143205] pci 0000:85:10.0: Adding to iommu group 58
[   12.148750] pci 0000:85:10.1: Adding to iommu group 58
[   12.154297] pci 0000:85:10.2: Adding to iommu group 58
[   12.159837] pci 0000:85:10.3: Adding to iommu group 58
[   12.165377] pci 0000:85:10.4: Adding to iommu group 58
[   12.170919] pci 0000:85:10.5: Adding to iommu group 58
[   12.176459] pci 0000:85:10.6: Adding to iommu group 58
[   12.181990] pci 0000:85:10.7: Adding to iommu group 58
[   12.187719] pci 0000:85:11.0: Adding to iommu group 59
[   12.193269] pci 0000:85:11.1: Adding to iommu group 59
[   12.198812] pci 0000:85:11.2: Adding to iommu group 59
[   12.204369] pci 0000:85:11.3: Adding to iommu group 59
[   12.210120] pci 0000:85:1d.0: Adding to iommu group 60
[   12.215671] pci 0000:85:1d.1: Adding to iommu group 60
[   12.221216] pci 0000:85:1d.2: Adding to iommu group 60
[   12.226758] pci 0000:85:1d.3: Adding to iommu group 60
[   12.232671] pci 0000:85:1e.0: Adding to iommu group 61
[   12.238238] pci 0000:85:1e.1: Adding to iommu group 61
[   12.243786] pci 0000:85:1e.2: Adding to iommu group 61
[   12.249327] pci 0000:85:1e.3: Adding to iommu group 61
[   12.254868] pci 0000:85:1e.4: Adding to iommu group 61
[   12.260408] pci 0000:85:1e.5: Adding to iommu group 61
[   12.265952] pci 0000:85:1e.6: Adding to iommu group 61
[   12.271457] pci 0000:86:00.0: Adding to iommu group 62
[   12.276972] pci 0000:87:00.0: Adding to iommu group 63
[   12.282483] pci 0000:88:00.0: Adding to iommu group 64
[   12.287994] pci 0000:89:00.0: Adding to iommu group 65
[   12.293498] pci 0000:ae:00.0: Adding to iommu group 66
[   12.299000] pci 0000:ae:01.0: Adding to iommu group 67
[   12.304499] pci 0000:ae:02.0: Adding to iommu group 68
[   12.310000] pci 0000:ae:03.0: Adding to iommu group 69
[   12.315687] pci 0000:ae:05.0: Adding to iommu group 70
[   12.321236] pci 0000:ae:05.2: Adding to iommu group 70
[   12.326790] pci 0000:ae:05.4: Adding to iommu group 70
[   12.332354] pci 0000:ae:08.0: Adding to iommu group 71
[   12.337913] pci 0000:ae:09.0: Adding to iommu group 72
[   12.343904] pci 0000:ae:0a.0: Adding to iommu group 73
[   12.349458] pci 0000:ae:0a.1: Adding to iommu group 73
[   12.355005] pci 0000:ae:0a.2: Adding to iommu group 73
[   12.360553] pci 0000:ae:0a.3: Adding to iommu group 73
[   12.366109] pci 0000:ae:0a.4: Adding to iommu group 73
[   12.371654] pci 0000:ae:0a.5: Adding to iommu group 73
[   12.377212] pci 0000:ae:0a.6: Adding to iommu group 73
[   12.382759] pci 0000:ae:0a.7: Adding to iommu group 73
[   12.388500] pci 0000:ae:0b.0: Adding to iommu group 74
[   12.394058] pci 0000:ae:0b.1: Adding to iommu group 74
[   12.399609] pci 0000:ae:0b.2: Adding to iommu group 74
[   12.405162] pci 0000:ae:0b.3: Adding to iommu group 74
[   12.411138] pci 0000:ae:0c.0: Adding to iommu group 75
[   12.416692] pci 0000:ae:0c.1: Adding to iommu group 75
[   12.422249] pci 0000:ae:0c.2: Adding to iommu group 75
[   12.427799] pci 0000:ae:0c.3: Adding to iommu group 75
[   12.433351] pci 0000:ae:0c.4: Adding to iommu group 75
[   12.438900] pci 0000:ae:0c.5: Adding to iommu group 75
[   12.444453] pci 0000:ae:0c.6: Adding to iommu group 75
[   12.450001] pci 0000:ae:0c.7: Adding to iommu group 75
[   12.455745] pci 0000:ae:0d.0: Adding to iommu group 76
[   12.461300] pci 0000:ae:0d.1: Adding to iommu group 76
[   12.466857] pci 0000:ae:0d.2: Adding to iommu group 76
[   12.472416] pci 0000:ae:0d.3: Adding to iommu group 76
[   12.478107] pci 0000:d7:05.0: Adding to iommu group 77
[   12.483667] pci 0000:d7:05.2: Adding to iommu group 77
[   12.489225] pci 0000:d7:05.4: Adding to iommu group 77
[   12.494848] pci 0000:d7:0e.0: Adding to iommu group 78
[   12.500404] pci 0000:d7:0e.1: Adding to iommu group 78
[   12.506031] pci 0000:d7:0f.0: Adding to iommu group 79
[   12.511586] pci 0000:d7:0f.1: Adding to iommu group 79
[   12.517210] pci 0000:d7:10.0: Adding to iommu group 80
[   12.522766] pci 0000:d7:10.1: Adding to iommu group 80
[   12.528566] pci 0000:d7:12.0: Adding to iommu group 81
[   12.534129] pci 0000:d7:12.1: Adding to iommu group 81
[   12.539684] pci 0000:d7:12.2: Adding to iommu group 81
[   12.545246] pci 0000:d7:12.4: Adding to iommu group 81
[   12.550803] pci 0000:d7:12.5: Adding to iommu group 81
[   12.556369] pci 0000:d7:15.0: Adding to iommu group 82
[   12.561987] pci 0000:d7:16.0: Adding to iommu group 83
[   12.567547] pci 0000:d7:16.4: Adding to iommu group 83
[   12.573114] pci 0000:d7:17.0: Adding to iommu group 84
[   12.578553] DMAR: Intel(R) Virtualization Technology for Directed I/O
[   12.777670] Initialise system trusted keyrings
[   12.782413] Key type blacklist registered
[   12.786776] workingset: timestamp_bits=36 max_order=27 bucket_order=0
[   12.795331] zbud: loaded
[   12.798722] pstore: using deflate compression
[   12.803673] Platform Keyring initialized
[   13.599036] tsc: Refined TSC clocksource calibration: 1995.312 MHz
[   13.605550] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x3985c314e25, max_idle_ns: 881590612270 ns
[   13.616183] clocksource: Switched to clocksource tsc
[   13.690373] NET: Registered protocol family 38
[   13.695130] Key type asymmetric registered
[   13.699531] Asymmetric key parser 'x509' registered
[   13.704729] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[   13.712822] io scheduler mq-deadline registered
[   13.723361] io scheduler kyber registered
[   13.727728] io scheduler bfq registered
[   13.732146] atomic64_test: passed for x86-64 platform with CX8 and with SSE
[   13.740068] pcieport 0000:00:1c.0: Signaling with IRQ 26
[   13.745715] pcieport 0000:00:1c.0: Slot #0 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl+ IbPresDis- LLActRep+
[   13.759357] pcieport 0000:00:1c.4: Signaling with IRQ 27
[   13.765340] pcieport 0000:17:00.0: Signaling with IRQ 28
[   13.771125] pcieport 0000:17:02.0: Signaling with IRQ 29
[   13.776926] pcieport 0000:3a:00.0: Signaling with IRQ 30
[   13.783079] pcieport 0000:85:00.0: Signaling with IRQ 32
[   13.788753] pcieport 0000:85:00.0: Slot #166 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.805114] pcieport 0000:85:01.0: Signaling with IRQ 33
[   13.810777] pcieport 0000:85:01.0: Slot #167 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.827112] pcieport 0000:85:02.0: Signaling with IRQ 34
[   13.832776] pcieport 0000:85:02.0: Slot #168 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.849100] pcieport 0000:85:03.0: Signaling with IRQ 35
[   13.854757] pcieport 0000:85:03.0: Slot #169 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.871098] pcieport 0000:ae:00.0: Signaling with IRQ 37
[   13.876759] pcieport 0000:ae:00.0: Slot #164 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.893038] pcieport 0000:ae:01.0: Signaling with IRQ 38
[   13.898698] pcieport 0000:ae:01.0: Slot #165 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.914967] pcieport 0000:ae:02.0: Signaling with IRQ 39
[   13.920620] pcieport 0000:ae:02.0: Slot #162 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.936887] pcieport 0000:ae:03.0: Signaling with IRQ 40
[   13.942545] pcieport 0000:ae:03.0: Slot #163 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlock- NoCompl- IbPresDis- LLActRep+ (with Cmd Compl erratum)
[   13.958717] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[   13.966519] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[   13.975024] ACPI: Power Button [PWRF]
[   13.983176] ERST: Error Record Serialization Table (ERST) support is initialized.
[   13.991327] pstore: Registered erst as persistent store backend
[   13.998432] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[   14.006570] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[   14.033921] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[   14.062550] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[   14.071317] Non-volatile memory driver v1.3
[   14.078389] rdac: device handler registered
[   14.082945] hp_sw: device handler registered
[   14.087524] emc: device handler registered
[   14.092105] alua: device handler registered
[   14.096630] libphy: Fixed MDIO Bus: probed
[   14.101129] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   14.107962] ehci-pci: EHCI PCI platform driver
[   14.112785] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[   14.119275] ohci-pci: OHCI PCI platform driver
[   14.124092] uhci_hcd: USB Universal Host Controller Interface driver
[   14.131072] xhci_hcd 0000:00:14.0: xHCI Host Controller
[   14.136652] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[   14.145713] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810
[   14.155382] xhci_hcd 0000:00:14.0: cache line size of 64 is not supported
[   14.162795] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.18
[   14.171596] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   14.179348] usb usb1: Product: xHCI Host Controller
[   14.184534] usb usb1: Manufacturer: Linux 4.18.0-193.14.3.rt13.67.el8_2.x86_64 xhci-hcd
[   14.193079] usb usb1: SerialNumber: 0000:00:14.0
[   14.198101] hub 1-0:1.0: USB hub found
[   14.202183] hub 1-0:1.0: 16 ports detected
[   14.207943] xhci_hcd 0000:00:14.0: xHCI Host Controller
[   14.213510] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[   14.221440] xhci_hcd 0000:00:14.0: Host supports USB 3.0  SuperSpeed
[   14.228147] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.18
[   14.236944] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   14.244695] usb usb2: Product: xHCI Host Controller
[   14.249880] usb usb2: Manufacturer: Linux 4.18.0-193.14.3.rt13.67.el8_2.x86_64 xhci-hcd
[   14.258407] usb usb2: SerialNumber: 0000:00:14.0
[   14.263406] hub 2-0:1.0: USB hub found
[   14.267474] hub 2-0:1.0: 10 ports detected
[   14.272321] usb: port power management may be unreliable
[   14.278538] usbcore: registered new interface driver usbserial_generic
[   14.285393] usbserial: USB Serial support registered for generic
[   14.291786] i8042: PNP: No PS/2 controller found.
[   14.296855] mousedev: PS/2 mouse device common for all mice
[   14.302835] rtc_cmos 00:00: RTC can wake from S4
[   14.308097] rtc_cmos 00:00: registered as rtc0
[   14.312883] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[   14.321589] hidraw: raw HID events driver (C) Jiri Kosina
[   14.327385] usbcore: registered new interface driver usbhid
[   14.333277] usbhid: USB HID core driver
[   14.338145] drop_monitor: Initializing network drop monitor service
[   14.344885] Initializing XFRM netlink socket
[   14.349773] NET: Registered protocol family 10
[   14.355342] Segment Routing with IPv6
[   14.359353] NET: Registered protocol family 17
[   14.364159] mpls_gso: MPLS GSO support
[   14.368236] core: Unable to init MCE device (rc: -5)
[   14.373646] core: Using 0 MCE banks
[   14.377507] microcode: sig=0x50654, pf=0x80, revision=0x2006906
[   14.377723] microcode: Microcode Update Driver: v2.2.
[   14.378246] resctrl: MB allocation detected
[   14.394379] AVX2 version of gcm_enc/dec engaged.
[   14.399321] AES CTR mode by8 optimization enabled
[   14.529009] usb 1-14: new high-speed USB device number 2 using xhci_hcd
[   14.661104] usb 1-14: New USB device found, idVendor=1604, idProduct=10c0, bcdDevice= 0.00
[   14.669902] usb 1-14: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   14.678078] hub 1-14:1.0: USB hub found
[   14.682235] hub 1-14:1.0: 4 ports detected
[   14.982600] sched_clock: Marking stable (14982595983, 0)->(18366535035, -3383939052)
[   14.991035] registered taskstats version 1
[   14.995472] Loading compiled-in X.509 certificates
[   15.083322] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: 4cc6a03fd8aaff4e2d5e76a818df72959cad3e3e'
[   15.094840] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80'
[   15.107305] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8'
[   15.118553] zswap: loaded using pool lzo/zbud
[   15.231542] Key type big_key registered
[   15.289882] Key type encrypted registered
[   15.294239] ima: No TPM chip found, activating TPM-bypass!
[   15.300063] ima: Allocated hash algorithm: sha1
[   15.304933] evm: Initialising EVM extended attributes:
[   15.310396] evm: security.selinux
[   15.314034] evm: security.ima
[   15.317347] evm: security.capability
[   15.321246] evm: HMAC attrs: 0x1
[   15.327113] rtc_cmos 00:00: setting system clock to 2020-09-23 11:01:50 UTC (1600858910)
[   15.338191] Freeing unused decrypted memory: 2040K
[   15.344584] Freeing unused kernel memory: 2328K
[   15.349441] Write protecting the kernel read-only data: 18432k
[   15.356903] Freeing unused kernel memory: 2020K
[   15.361959] Freeing unused kernel memory: 304K
[   15.367008] usb 1-14.1: new high-speed USB device number 3 using xhci_hcd
[   15.453888] usb 1-14.1: New USB device found, idVendor=1604, idProduct=10c0, bcdDevice= 0.00
[   15.471629] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked executable. Please remove executable permission bits. Proceeding anyway.
[   15.471633] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[   15.515978] usb 1-14.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   15.523428] systemd[1]: systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[   15.535048] systemd[1]: Detected architecture x86-64.
[   15.535050] systemd[1]: Running in initial RAM disk.
[   15.557245] hub 1-14.1:1.0: USB hub found
[   15.562590] hub 1-14.1:1.0: 4 ports detected
[   15.579032] systemd[1]: No hostname configured.
[   15.583897] systemd[1]: Set hostname to <localhost>.
[   15.589217] systemd[1]: Initializing machine ID from random generator.
[   15.632014] usb 1-14.4: new high-speed USB device number 4 using xhci_hcd
[   15.682353] systemd[1]: Reached target Timers.
[   15.698140] systemd[1]: Listening on Journal Socket (/dev/log).
[   15.711046] systemd[1]: Reached target Initrd Root Device.
[   15.715107] usb 1-14.4: New USB device found, idVendor=1604, idProduct=10c0, bcdDevice= 0.00
[   15.715109] usb 1-14.4: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   15.715633] hub 1-14.4:1.0: USB hub found
[   15.715655] hub 1-14.4:1.0: 4 ports detected
[   15.749082] systemd[1]: Listening on udev Kernel Socket.
[   15.842439] Loading iSCSI transport class v2.0-870.
[   15.857226] fuse: init (API version 7.31)
[   15.963062] systemd[1]: Started Apply Kernel Variables.
[   15.972721] systemd[1]: Started Create Static Device Nodes in /dev.
[   15.985437] systemd[1]: Started dracut ask for additional cmdline parameters.
[   16.002180] systemd[1]: Starting dracut cmdline hook...
[   16.191464] systemd[1]: Started Journal Service.
[   16.191517] systemd-modules-load[1077]: Inserted module 'fuse'
[   16.192367] dracut-cmdline[1105]: dracut-45.82.202009040230-0 dracut-049-70.git20200228.el8
[   16.192427] dracut-cmdline[1105]: Using kernel command line parameters: rd.driver.pre=dm_multipath BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/vmlinuz-4.18.0-193.14.3.rt13.67.el8_2.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.0/rhcos/b6d8f1620a8d828e74f076ee8ffeba009c9dc3a88da60d17c41cf1bac6f66757/0 ignition.platform.id=openstack nohz=on nosoftlockup skew_tick=1 intel_pstate=disable intel_iommu=on iommu=pt default_hugepagesz=1G nmi_watchdog=0 audit=0 mce=off processor.max_cstate=1 idle=poll intel_idle.max_cstate=0 nosmt tsc=reliable rcu_nocbs=5-60 tuned.non_isolcpus=0000001f
[   16.223194] iscsi: registered transport (tcp)
[   16.259909] iscsi: registered transport (qla4xxx)
[   16.260090] QLogic iSCSI HBA Driver
[   16.266414] libcxgbi:libcxgbi_init_module: Chelsio iSCSI driver library libcxgbi v0.9.1-ko (Apr. 2015)
[   16.296307] Chelsio T4-T6 iSCSI Driver cxgb4i v0.9.5-ko (Apr. 2015)
[   16.296320] iscsi: registered transport (cxgb4i)
[   16.303000] cnic: QLogic cnicDriver v2.5.22 (July 20, 2015)
[   16.307435] QLogic NetXtreme II iSCSI Driver bnx2i v2.7.10.1 (Jul 16, 2014)
[   16.307447] iscsi: registered transport (bnx2i)
[   16.335091] iscsi: registered transport (be2iscsi)
[   16.340216] In beiscsi_module_init, tt=00000000c7380522
[   16.389399] systemd[1]: Started dracut cmdline hook.
[   16.402194] systemd[1]: Starting dracut pre-udev hook...
[   16.414422] tuned: setting workqueue CPU mask to 0000001f
[   16.453263] device-mapper: uevent: version 1.0.3
[   16.458275] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com
[   16.532563] systemd[1]: Started dracut pre-udev hook.
[   16.545435] systemd[1]: Starting udev Kernel Device Manager...
[   16.794785] systemd-udevd[1401]: Specified group 'tty' unknown
[   16.801094] systemd-udevd[1401]: Specified group 'kmem' unknown
[   16.808706] systemd-udevd[1401]: Specified group 'input' unknown
[   16.817486] systemd-udevd[1401]: Specified group 'video' unknown
[   16.824863] systemd-udevd[1401]: Specified group 'render' unknown
[   16.831720] systemd-udevd[1401]: Specified group 'audio' unknown
[   16.838095] systemd-udevd[1401]: Specified group 'lp' unknown
[   16.845702] systemd-udevd[1401]: Specified group 'disk' unknown
[   16.852983] systemd-udevd[1401]: Specified group 'kvm' unknown
[   16.860182] dracut-pre-trigger[1418]: rd.md=0: removing MD RAID activation
[   16.867409] systemd[1]: Started udev Kernel Device Manager.
[   16.873323] systemd[1]: Starting dracut pre-trigger hook...
[   16.969304] systemd[1]: Started dracut pre-trigger hook.
[   16.981946] systemd[1]: Starting udev Coldplug all Devices...
[   17.022173] systemd-udevd[1401]: Specified group 'tty' unknown
[   17.028352] systemd-udevd[1401]: Specified group 'kmem' unknown
[   17.035306] systemd-udevd[1401]: Specified group 'input' unknown
[   17.042206] systemd-udevd[1401]: Specified group 'video' unknown
[   17.048578] systemd-udevd[1401]: Specified group 'render' unknown
[   17.055179] systemd-udevd[1401]: Specified group 'audio' unknown
[   17.061541] systemd-udevd[1401]: Specified group 'lp' unknown
[   17.067627] systemd-udevd[1401]: Specified group 'disk' unknown
[   17.073882] systemd-udevd[1401]: Specified group 'kvm' unknown
[   17.109737] systemd[1]: Mounting Kernel Configuration File System...
[   17.123507] systemd[1]: Mounted Kernel Configuration File System.
[   17.786214] libata version 3.00 loaded.
[   17.831646] megasas: 07.710.50.00-rc1
[   17.838684] megaraid_sas 0000:18:00.0: BAR:0x1  BAR's base_addr(phys):0x00000000a2b00000  mapped virt_addr:0x0000000094234e07
[   17.838687] megaraid_sas 0000:18:00.0: FW now in Ready state
[   17.844654] megaraid_sas 0000:18:00.0: 63 bit DMA mask and 32 bit consistent mask
[   17.867783] megaraid_sas 0000:18:00.0: Using iommu dma mapping
[   17.879686] megaraid_sas 0000:18:00.0: 32bit DMA uses non-identity mapping
[   17.894190] megaraid_sas 0000:18:00.0: firmware supports msix    : (96)
[   17.900949] ahci 0000:00:11.5: version 3.0
[   17.908017] ahci 0000:00:11.5: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[   17.916787] ahci 0000:00:11.5: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst 
[   17.927876] mlx5_core 0000:19:00.0: firmware version: 14.26.6000
[   17.934226] mlx5_core 0000:19:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[   18.042946] megaraid_sas 0000:18:00.0: requested/available msix 41/41
[   18.049710] megaraid_sas 0000:18:00.0: current msix/online cpus  : (41/40)
[   18.056229] nvme nvme0: pci function 0000:86:00.0
[   18.062887] nvme nvme1: pci function 0000:87:00.0
[   18.070979] nvme nvme2: pci function 0000:88:00.0
[   18.080041] nvme nvme3: pci function 0000:89:00.0
[   18.098621] megaraid_sas 0000:18:00.0: RDPQ mode : (disabled)
[   18.102404] scsi host1: ahci
[   18.116450] megaraid_sas 0000:18:00.0: Current firmware supports maximum commands: 928    LDIO threshold: 237
[   18.129475] scsi host2: ahci
[   18.142870] megaraid_sas 0000:18:00.0: Configured max firmware commands: 927
[   18.145205] scsi host3: ahci
[   18.162055] scsi host4: ahci
[   18.173623] scsi host5: ahci
[   18.181732] megaraid_sas 0000:18:00.0: Performance mode :Latency
[   18.191292] scsi host6: ahci
[   18.200352] megaraid_sas 0000:18:00.0: FW supports sync cache    : No
[   18.209794] megaraid_sas 0000:18:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[   18.213014] ata1: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80100 irq 44
[   18.213015] ata2: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80180 irq 44
[   18.213017] ata3: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80200 irq 44
[   18.213018] ata4: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80280 irq 44
[   18.213020] ata5: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80300 irq 44
[   18.213021] ata6: SATA max UDMA/133 abar m524288@0x92c80000 port 0x92c80380 irq 44
[   18.254124] mlx5_core 0000:19:00.0: E-Switch: Total vports 7, per vport: max uc(1024) max mc(16384)
[   18.282881] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0xff impl SATA mode
[   18.291613] ahci 0000:00:17.0: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst 
[   18.341125] scsi host7: ahci
[   18.343369] systemd[1]: Started udev Coldplug all Devices.
[   18.350407] scsi host8: ahci
[   18.353934] scsi host9: ahci
[   18.358207] scsi host10: ahci
[   18.362606] scsi host11: ahci
[   18.367041] mlx5_core 0000:19:00.0: Port module event: module 1, Cable plugged
[   18.369357] nvme nvme2: 80/0/0 default/read/poll queues
[   18.370028] nvme nvme1: 80/0/0 default/read/poll queues
[   18.374072] nvme nvme0: 80/0/0 default/read/poll queues
[   18.375275] nvme nvme3: 80/0/0 default/read/poll queues
[   18.381624] systemd[1]: Starting udev Wait for Complete Device Initialization...
[   18.409579] scsi host12: ahci
[   18.414601] scsi host13: ahci
[   18.415224] mlx5_core 0000:19:00.1: firmware version: 14.26.6000
[   18.415278] mlx5_core 0000:19:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[   18.470374] scsi host14: ahci
[   18.473713] ata7: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00100 irq 96
[   18.481816] ata8: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00180 irq 96
[   18.489912] ata9: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00200 irq 96
[   18.498035] ata10: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00280 irq 96
[   18.506225] ata11: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00300 irq 96
[   18.514415] ata12: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00380 irq 96
[   18.522598] ata13: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00400 irq 96
[   18.524155] ata6: SATA link down (SStatus 0 SControl 300)
[   18.524263] ata1: SATA link down (SStatus 0 SControl 300)
[   18.524297] ata2: SATA link down (SStatus 0 SControl 300)
[   18.524332] ata3: SATA link down (SStatus 0 SControl 300)
[   18.524367] ata5: SATA link down (SStatus 0 SControl 300)
[   18.524401] ata4: SATA link down (SStatus 0 SControl 300)
[   18.565036] ata14: SATA max UDMA/133 abar m524288@0x92c00000 port 0x92c00480 irq 96
[   18.654008] megaraid_sas 0000:18:00.0: FW provided supportMaxExtLDs: 0   max_lds: 32
[   18.662131] megaraid_sas 0000:18:00.0: controller type   : iMR(0MB)
[   18.668524] megaraid_sas 0000:18:00.0: Online Controller Reset(OCR)  : Enabled
[   18.675988] megaraid_sas 0000:18:00.0: Secure JBOD support   : No
[   18.682222] megaraid_sas 0000:18:00.0: NVMe passthru support : No
[   18.688651] megaraid_sas 0000:18:00.0: FW provided TM TaskAbort/Reset timeout    : 0 secs/0 secs
[   18.698248] megaraid_sas 0000:18:00.0: JBOD sequence map support : No
[   18.705005] megaraid_sas 0000:18:00.0: PCI Lane Margining support    : No
[   18.714973] mlx5_core 0000:19:00.1: E-Switch: Total vports 7, per vport: max uc(1024) max mc(16384)
[   18.736895] megaraid_sas 0000:18:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[   18.747034] megaraid_sas 0000:18:00.0: INIT adapter done
[   18.751034] mlx5_core 0000:19:00.1: Port module event: module 0, Cable plugged
[   18.760636] megaraid_sas 0000:18:00.0: JBOD sequence map is disabled megasas_setup_jbod_map 5653
[   18.771578] megaraid_sas 0000:18:00.0: pci id        : (0x1000)/(0x005f)/(0x1028)/(0x1f4b)
[   18.780072] megaraid_sas 0000:18:00.0: unevenspan support    : yes
[   18.786481] megaraid_sas 0000:18:00.0: firmware crash dump   : no
[   18.792922] megaraid_sas 0000:18:00.0: JBOD sequence map : disabled
[   18.799568] scsi host0: Avago SAS based MegaRAID driver
[   18.808672] mlx5_core 0000:3b:00.0: firmware version: 14.26.6000
[   18.815021] mlx5_core 0000:3b:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[   18.884136] ata9: SATA link down (SStatus 0 SControl 300)
[   18.889879] ata13: SATA link down (SStatus 0 SControl 300)
[   18.891669] scsi 0:2:0:0: Direct-Access     DELL     PERC H330 Mini   4.29 PQ: 0 ANSI: 5
[   18.904361] ata14: SATA link down (SStatus 0 SControl 300)
[   18.910192] ata8: SATA link down (SStatus 0 SControl 300)
[   18.915953] ata10: SATA link down (SStatus 0 SControl 300)
[   18.921785] ata11: SATA link down (SStatus 0 SControl 300)
[   18.927619] ata7: SATA link down (SStatus 0 SControl 300)
[   18.933366] ata12: SATA link down (SStatus 0 SControl 300)
[   18.981201] systemd-udevd[1481]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   19.101506] mlx5_core 0000:3b:00.0: E-Switch: Total vports 2, per vport: max uc(1024) max mc(16384)
[   19.138339] mlx5_core 0000:3b:00.0: Port module event: module 0, Cable plugged
[   19.160282] mlx5_core 0000:3b:00.1: firmware version: 14.26.6000
[   19.162251] scsi 0:2:0:0: Attached scsi generic sg0 type 0
[   19.168073] sd 0:2:0:0: [sda] 936640512 512-byte logical blocks: (480 GB/447 GiB)
[   19.168075] sd 0:2:0:0: [sda] 4096-byte physical blocks
[   19.168249] sd 0:2:0:0: [sda] Write Protect is off
[   19.168250] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08
[   19.168322] sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA
[   19.172635]  sda: sda1 sda2 sda3 sda4 sda5
[   19.174184] sd 0:2:0:0: [sda] Attached SCSI disk
[   19.209734] mlx5_core 0000:3b:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[   19.257587] systemd[1]: Found device PERC_H330_Mini crypt_rootfs.
[   19.478543] mlx5_core 0000:3b:00.1: E-Switch: Total vports 2, per vport: max uc(1024) max mc(16384)
[   19.493976] mlx5_core 0000:3b:00.1: Port module event: module 1, Cable plugged
[   19.516558] mlx5_core 0000:19:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[   19.752284] mlx5_core 0000:19:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[   19.984647] mlx5_core 0000:3b:00.0: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[   20.222483] mlx5_core 0000:3b:00.1: MLX5E: StrdRq(0) RqSz(1024) StrdSz(256) RxCqeCmprss(0)
[   20.474974] mlx5_core 0000:19:00.0 eno1: renamed from eth0
[   20.475115] systemd-udevd[1480]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   20.507274] systemd-udevd[1494]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   20.507988] mlx5_core 0000:19:00.1 eno2: renamed from eth1
[   20.512175] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0
[   20.536283] mlx5_core 0000:3b:00.1 ens1f1: renamed from eth3
[   20.564667] systemd-udevd[1478]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   20.606547] systemd-udevd[1479]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
[   20.608495] mlx5_core 0000:3b:00.0 ens1f0: renamed from eth2
[   22.273544] systemd[1]: Started udev Wait for Complete Device Initialization.
[   22.290365] systemd[1]: Starting Device-Mapper Multipath Device Controller...
[   22.310573] systemd[1]: Started Device-Mapper Multipath Device Controller.
[   22.325652] multipathd[2184]: --------start up--------
[   22.331163] multipathd[2184]: read /etc/multipath.conf
[   22.336710] multipathd[2184]: /etc/multipath.conf does not exist, blacklisting all devices.
[   22.346657] multipathd[2184]: You can run "/sbin/mpathconf --enable" to create
[   22.355274] multipathd[2184]: /etc/multipath.conf. See man mpathconf(8) for more details
[   22.363957] multipathd[2184]: path checkers start up
[   22.369296] multipathd[2184]: /etc/multipath.conf does not exist, blacklisting all devices.
[   22.379280] multipathd[2184]: You can run "/sbin/mpathconf --enable" to create
[   22.387891] multipathd[2184]: /etc/multipath.conf. See man mpathconf(8) for more details
[   22.397448] systemd[1]: Starting Open-iSCSI...
[   22.403050] iscsid[2185]: iscsid: can't open InitiatorName configuration file /etc/iscsi/initiatorname.iscsi
[   22.414622] iscsid[2185]: iscsid: Warning: InitiatorName file /etc/iscsi/initiatorname.iscsi does not exist or does not contain a properly formatted InitiatorName. If using software iscsi (iscsi_tcp or ib_iser) or partial offload (bnx2i or cxgbi iscsi), you may not be able to log into or discover targets. Please create a file /etc/iscsi/initiatorname.iscsi that contains a sting with the format: InitiatorName=iqn.yyyy-mm.<reversed domain name>[:identifier].
[   22.457379] iscsid[2185]: Example: InitiatorName=iqn.2001-04.com.redhat:fc6.
[   22.465204] iscsid[2185]: If using hardware iscsi like qla4xxx this message can be ignored.
[   22.474151] iscsid[2185]: iscsid: can't open InitiatorAlias configuration file /etc/iscsi/initiatorname.iscsi
[   22.484731] systemd[1]: Reached target Local File Systems (Pre).
[   22.491886] systemd[1]: Reached target Local File Systems.
[   22.498818] systemd[1]: Starting Create Volatile Files and Directories...
[   22.507093] systemd-tmpfiles[2192]: [/usr/lib/tmpfiles.d/systemd.conf:11] Unknown group 'utmp'.
[   22.516906] systemd[1]: Started Open-iSCSI.
[   22.526567] systemd[1]: Started Create Volatile Files and Directories.
[   22.541358] systemd[1]: Reached target System Initialization.
[   22.554081] systemd[1]: Reached target Basic System.
[   22.566914] systemd[1]: Starting CoreOS LUKS Opener...
[   22.577913] systemd[1]: Starting dracut initqueue hook...
[   22.591538] systemd[1]: Started dracut initqueue hook.
[   22.603244] systemd[1]: Reached target Remote File Systems (Pre).
[   22.616098] systemd[1]: Reached target Remote File Systems.
[   22.628875] systemd[1]: Starting dracut pre-mount hook...
[   22.641075] systemd[1]: Started dracut pre-mount hook.
[   22.684413] coreos-cryptfs[2194]: coreos-cryptfs: /dev/sda4 uses the 'cipher_null-ecb'
[   22.692898] coreos-cryptfs[2194]: coreos-cryptfs: /dev/sda4 is not encrypted. Device will be mounted as device-mapper linear target.
[   22.738430] coreos-cryptfs[2194]: NOCHANGE: partition 4 is size 935456768. it cannot be grown
[   22.767575] systemd[1]: Found device /dev/disk/by-label/root.
[   22.780515] systemd[1]: Started CoreOS LUKS Opener.
[   22.792367] systemd[1]: Reached target Ignition Subsequent Boot Disk Setup.
[   22.807067] systemd[1]: Reached target Subsequent (Not Ignition) boot complete.
[   22.822936] systemd[1]: Starting CoreOS: Mount (subsequent) /sysroot...
[   22.959813] SGI XFS with ACLs, security attributes, no debug enabled
[   22.971395] XFS (dm-0): Mounting V5 Filesystem
[   22.993413] XFS (dm-0): Ending clean mount
[   23.318377] systemd[1]: Started CoreOS: Mount (subsequent) /sysroot.
[   23.334151] systemd[1]: Starting OSTree Prepare OS/...
[   23.349559] ostree-prepare-root[2286]: Resolved OSTree target to: /sysroot/ostree/deploy/rhcos/deploy/f51ff069f7ae96312b0281e06acea9cffb200ff72df00d437c9b5d259136ebdc.0
[   23.365451] systemd[1]: Started OSTree Prepare OS/.
[   23.377217] systemd[1]: Reached target Initrd Root File System.
[   23.390813] systemd[1]: Starting Reload Configuration from the Real Root...
[   23.422574] systemd[1]: Reloading.
[   23.426325] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked executable. Please remove executable permission bits. Proceeding anyway.
[   23.443897] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[   23.613408] systemd[1]: Started Reload Configuration from the Real Root.
[   23.628183] systemd[1]: Reached target Initrd File Systems.
[   23.640164] systemd[1]: Reached target Initrd Default Target.
[   23.653981] systemd[1]: Starting dracut pre-pivot and cleanup hook...
[   23.741221] dracut-pre-pivot[2355]: cp: cannot stat '/tmp/ifcfg/*': No such file or directory
[   23.858744] dracut-pre-pivot[2355]: Sep 23 11:01:59 | /etc/multipath.conf does not exist, blacklisting all devices.
[   23.888101] dracut-pre-pivot[2355]: Sep 23 11:01:59 | You can run "/sbin/mpathconf --enable" to create
[   23.915085] dracut-pre-pivot[2355]: Sep 23 11:01:59 | /etc/multipath.conf. See man mpathconf(8) for more details
[   23.944189] systemd[1]: Started dracut pre-pivot and cleanup hook.
[   23.952108] systemd[1]: Starting Cleaning Up and Shutting Down Daemons...
[   23.960200] systemd[1]: Stopped dracut pre-pivot and cleanup hook.
[   23.967777] systemd[1]: Stopped dracut pre-mount hook.
[   23.973262] systemd[1]: Stopped target Initrd Default Target.
[   23.986069] systemd[1]: Stopped target Subsequent (Not Ignition) boot complete.
[   24.001047] systemd[1]: Stopped target Ignition Subsequent Boot Disk Setup.
[   24.016044] systemd[1]: Stopped target Initrd Root Device.
[   24.029084] systemd[1]: Stopped CoreOS: Mount (subsequent) /sysroot.
[   24.043520] systemd[1]: Stopped CoreOS LUKS Opener.
[   24.055099] systemd[1]: Stopped target Basic System.
[   24.067067] systemd[1]: Stopped target Paths.
[   24.077059] systemd[1]: Stopped target Sockets.
[   24.087059] systemd[1]: Stopped target Timers.
[   24.097057] systemd[1]: Stopped target Remote File Systems.
[   24.109059] systemd[1]: Stopped target Remote File Systems (Pre).
[   24.122478] systemd[1]: Stopped dracut initqueue hook.
[   24.134162] systemd[1]: Stopping Open-iSCSI...
[   24.139182] iscsid[2185]: iscsid: iscsid shutting down.
[   24.151307] systemd[1]: Stopped target System Initialization.
[   24.164118] systemd[1]: Stopped target Swap.
[   24.174549] systemd[1]: Stopped Create Volatile Files and Directories.
[   24.189081] systemd[1]: Stopped target Local File Systems.
[   24.201071] systemd[1]: Stopped target Local File Systems (Pre).
[   24.214536] systemd[1]: Stopped Apply Kernel Variables.
[   24.226578] systemd[1]: Stopped Load Kernel Modules.
[   24.238064] systemd[1]: Stopped target Local Encrypted Volumes.
[   24.255055] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
[   24.272047] systemd[1]: Stopped target Slices.
[   24.283303] systemd[1]: Stopped Open-iSCSI.
[   24.294452] systemd[1]: Stopping Device-Mapper Multipath Device Controller...
[   24.307293] multipathd[2396]: Sep 23 11:01:59 | /etc/multipath.conf does not exist, blacklisting all devices.
[   24.317968] multipathd[2396]: Sep 23 11:01:59 | You can run "/sbin/mpathconf --enable" to create
[   24.328729] multipathd[2396]: Sep 23 11:01:59 | /etc/multipath.conf. See man mpathconf(8) for more details
[   24.339738] multipathd[2396]: ok
[   24.344306] multipathd[2184]: --------shut down-------
[   24.349935] systemd[1]: Stopping iSCSI UserSpace I/O driver...
[   24.357575] systemd[1]: Closed Open-iSCSI iscsid Socket.
[   24.368581] systemd[1]: Stopped iSCSI UserSpace I/O driver.
[   24.381804] systemd[1]: Started Cleaning Up and Shutting Down Daemons.
[   24.396615] systemd[1]: Stopped Device-Mapper Multipath Device Controller.
[   24.413489] systemd[1]: Stopped udev Wait for Complete Device Initialization.
[   24.429525] systemd[1]: Stopped udev Coldplug all Devices.
[   24.442565] systemd[1]: Stopped dracut pre-trigger hook.
[   24.455131] systemd[1]: Stopping udev Kernel Device Manager...
[   24.468444] systemd[1]: Closed Open-iSCSI iscsiuio Socket.
[   24.481548] systemd[1]: Stopped udev Kernel Device Manager.
[   24.494688] systemd[1]: Stopped Create Static Device Nodes in /dev.
[   24.509585] systemd[1]: Stopped Create list of required static device nodes for the current kernel.
[   24.528425] systemd[1]: Stopped dracut pre-udev hook.
[   24.540535] systemd[1]: Stopped dracut cmdline hook.
[   24.552511] systemd[1]: Stopped dracut ask for additional cmdline parameters.
[   24.568438] systemd[1]: Closed udev Kernel Socket.
[   24.579439] systemd[1]: Closed udev Control Socket.
[   24.591890] systemd[1]: Starting Cleanup udevd DB...
[   24.626811] systemd[1]: Started Cleanup udevd DB.
[   24.637153] systemd[1]: Reached target Switch Root.
[   24.649919] systemd[1]: Starting Switch Root...
[   24.683208] systemd[1]: Switching root.
[   24.736002] systemd-journald[1076]: Received SIGTERM from PID 1 (systemd).
[   24.781838] printk: systemd: 22 output lines suppressed due to ratelimiting
[   25.374207] SELinux:  policy capability network_peer_controls=1
[   25.380433] SELinux:  policy capability open_perms=1
[   25.385706] SELinux:  policy capability extended_socket_class=1
[   25.391932] SELinux:  policy capability always_check_network=0
[   25.398075] SELinux:  policy capability cgroup_seclabel=1
[   25.403776] SELinux:  policy capability nnp_nosuid_transition=1
[   25.470922] systemd[1]: Successfully loaded SELinux policy in 642.637ms.
[   25.536209] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 31.282ms.
[   25.545912] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked executable. Please remove executable permission bits. Proceeding anyway.
[   25.561819] systemd[1]: Configuration file /etc/systemd/system.conf.d/setAffinity.conf is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[   25.582742] systemd[1]: systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[   25.617039] systemd[1]: Detected architecture x86-64.
[   25.878416] systemd[1]: systemd-journald.service: Consumed 0 CPU time
[   25.886210] systemd[1]: Stopped Switch Root.
[   25.897059] systemd[1]: initrd-switch-root.service: Consumed 0 CPU time
[   25.904308] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[   25.915271] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[   25.924723] systemd[1]: Stopped Journal Service.
[   25.935022] systemd[1]: systemd-journald.service: Consumed 0 CPU time
[   25.942909] systemd[1]: Starting Journal Service...
[   25.954195] systemd[1]: Starting Run update-ca-trust...
[   25.965624] systemd[1]: Listening on multipathd control socket.
[   25.979621] systemd[1]: Created slice system-getty.slice.
[   25.992065] systemd[1]: Reached target Host and Network Name Lookups.
[   26.008244] systemd[1]: Mounting Kernel Debug File System...
[   26.022586] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   26.042566] systemd[1]: Listening on udev Kernel Socket.
[   26.055180] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   26.070049] systemd[1]: Reached target Remote Encrypted Volumes.
[   26.083273] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   26.101666] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[   26.117618] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[   26.132030] systemd[1]: Reached target Remote File Systems.
[   26.145581] systemd[1]: Created slice system-serial\x2dgetty.slice.
[   26.163701] systemd[1]: Listening on Process Core Dump Socket.
[   26.177607] systemd[1]: Created slice system-sshd\x2dkeygen.slice.
[   26.191028] systemd[1]: Stopped target Switch Root.
[   26.203025] systemd[1]: Stopped target Initrd Root File System.
[   26.216824] systemd[1]: Stopped OSTree Prepare OS/.
[   26.229053] systemd[1]: ostree-prepare-root.service: Consumed 0 CPU time
[   26.236603] systemd[1]: Created slice User and Session Slice.
[   26.249044] systemd[1]: Reached target Slices.
[   26.260338] systemd[1]: Mounting Huge Pages File System...
[   26.271880] systemd[1]: Listening on LVM2 poll daemon socket.
[   26.286106] systemd[1]: Listening on RPCbind Server Activation Socket.
[   26.301042] systemd[1]: Reached target RPC Port Mapper.
[   26.313038] systemd[1]: Stopped target Initrd File Systems.
[   26.326198] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   26.342044] systemd[1]: Reached target Local Encrypted Volumes.
[   26.355037] systemd[1]: Reached target Swap.
[   26.366227] systemd[1]: Mounting Temporary Directory (/tmp)...
[   26.380296] systemd[1]: Mounting POSIX Message Queue File System...
[   26.395237] systemd[1]: Starting Create System Users...
[   26.408194] systemd[1]: Starting Rebuild Hardware Database...
[   26.420733] systemd[1]: Listening on udev Control Socket.
[   26.434220] systemd[1]: Starting udev Coldplug all Devices...
[   26.448122] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[   26.468287] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[   26.491396] systemd[1]: Starting Load Kernel Modules...
[   26.510725] systemd[1]: sysroot-usr.mount: Consumed 0 CPU time
[   26.517711] systemd[1]: sysroot-sysroot.mount: Consumed 0 CPU time
[   26.524507] systemd[1]: Started Journal Service.
[   27.073268] ACPI Error: No handler for Region [SYSI] (00000000e64fcbad) [IPMI] (20190703/evregion-132)
[   27.083184] ACPI Error: Region IPMI (ID=7) has no handler (20190703/exfldio-265)
[   27.091213] ACPI Error: Aborting method \_SB.PMI0._GHL due to previous error (AE_NOT_EXIST) (20190703/psparse-531)
[   27.102159] ACPI Error: Aborting method \_SB.PMI0._PMC due to previous error (AE_NOT_EXIST) (20190703/psparse-531)
[   27.113081] ACPI Error: AE_NOT_EXIST, Evaluating _PMC (20190703/power_meter-756)
[   28.935175] IPMI message handler: version 39.2
[   28.950867] ipmi device interface
[   28.970687] ipmi_si: IPMI System Interface driver
[   28.976357] ipmi_si dmi-ipmi-si.0: probing via SMBIOS
[   28.981760] ipmi_platform: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10
[   28.989659] ipmi_si: Adding SMBIOS-specified kcs state machine
[   28.996356] ipmi_si IPI0001:00: probing via ACPI
[   29.001348] ipmi_si IPI0001:00: [io  0x0ca8] regsize 1 spacing 4 irq 10
[   29.008290] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI
[   29.017549] ipmi_si: Adding ACPI-specified kcs state machine
[   29.026883] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10
[   29.145046] ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed.
[   29.180634] ipmi_si IPI0001:00: Using irq 10
[   29.217924] ipmi_si IPI0001:00: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20)
[   29.344664] ipmi_si IPI0001:00: IPMI kcs interface initialized
[   29.366133] ipmi_ssif: IPMI SSIF Interface driver
[   29.579947] wmi_bus wmi_bus-PNP0C14:00: WQBC data block query control method not found
[   29.655506] lpc_ich 0000:00:1f.0: I/O space for ACPI uninitialized
[   29.661991] lpc_ich 0000:00:1f.0: No MFD cells added
[   29.672270] i801_smbus 0000:00:1f.4: SPD Write Disable is set
[   29.679045] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[   30.112694] mgag200 0000:03:00.0: vgaarb: deactivate vga console
[   30.127854] Console: switching to colour dummy device 80x25
[   30.147816] [TTM] Zone  kernel: Available graphics memory: 197382756 KiB
[   30.154520] [TTM] Zone   dma32: Available graphics memory: 2097152 KiB
[   30.154521] [TTM] Initializing pool allocator
[   30.161189] [TTM] Initializing DMA pool allocator
[   30.220920] fbcon: mgag200drmfb (fb0) is primary device
[   30.225172] Console: switching to colour frame buffer device 128x48
[   30.362783] iscsi: registered transport (iser)
[   30.467005] RPC: Registered named UNIX socket transport module.
[   30.467006] RPC: Registered udp transport module.
[   30.467007] RPC: Registered tcp transport module.
[   30.467007] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   30.509106] RPC: Registered rdma transport module.
[   30.509108] RPC: Registered rdma backchannel transport module.
[   30.532594] input: PC Speaker as /devices/platform/pcspkr/input/input1
[   30.661060] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 655360 ms ovfl timer
[   30.661061] RAPL PMU: hw unit of domain package 2^-14 Joules
[   30.661062] RAPL PMU: hw unit of domain dram 2^-16 Joules
[   30.687228] mgag200 0000:03:00.0: fb0: mgag200drmfb frame buffer device
[   30.691045] [drm] Initialized mgag200 1.0.0 20110418 for 0000:03:00.0 on minor 0
[   30.847077] EDAC MC0: Giving out device to module skx_edac controller Skylake Socket#0 IMC#0: DEV 0000:3a:0a.0 (INTERRUPT)
[   30.858546] EDAC MC1: Giving out device to module skx_edac controller Skylake Socket#0 IMC#1: DEV 0000:3a:0c.0 (INTERRUPT)
[   30.869971] EDAC MC2: Giving out device to module skx_edac controller Skylake Socket#1 IMC#0: DEV 0000:ae:0a.0 (INTERRUPT)
[   30.881423] EDAC MC3: Giving out device to module skx_edac controller Skylake Socket#1 IMC#1: DEV 0000:ae:0c.0 (INTERRUPT)
[   32.818073] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3)
[   33.027387] iTCO_vendor_support: vendor-support=0
[   33.045266] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   33.051952] iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
[   33.076601] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[   33.195713] intel_rapl_common: Found RAPL domain package
[   33.207591] intel_rapl_common: Found RAPL domain dram
[   33.219095] intel_rapl_common: DRAM domain energy unit 15300pj
[   33.231568] intel_rapl_common: Found RAPL domain package
[   33.243301] intel_rapl_common: Found RAPL domain dram
[   33.254633] intel_rapl_common: DRAM domain energy unit 15300pj
[   33.360461] systemd-journald[2440]: Received request to flush runtime journal from PID 1
[   33.478513] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[   33.565712] Rounding down aligned max_sectors from 4294967295 to 4294967288
[   33.579312] db_root: cannot open: /etc/target
[   34.620694] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   34.945496] mlx5_core 0000:19:00.0 eno1: Link up
[   34.960662] IPv6: ADDRCONF(NETDEV_UP): eno1: link is not ready
[   34.974906] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
[   34.994322] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[   35.318916] mlx5_core 0000:19:00.1 eno2: Link up
[   35.333879] IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
[   35.353429] IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready
[   35.677767] mlx5_core 0000:3b:00.0 ens1f0: Link up
[   35.693017] IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready
[   35.714766] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready
[   36.041849] mlx5_core 0000:3b:00.1 ens1f1: Link up
[   36.057104] IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready
[   36.071531] IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
[   36.084550] IPv6: ADDRCONF(NETDEV_CHANGE): ens1f0: link becomes ready
[   36.097646] IPv6: ADDRCONF(NETDEV_CHANGE): ens1f1: link becomes ready
[   36.951796] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   37.663497] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   64.517489] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.261221] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.280181] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.286321] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.289528] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.311557] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.328187] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.395243] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.721449] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.866854] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   73.898766] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.185512] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.198991] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.203873] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.232002] openvswitch: Open vSwitch switching datapath
[   74.308058] device ovs-system entered promiscuous mode
[   74.457258] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.457318] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   74.547231] device genev_sys_6081 entered promiscuous mode
[   74.558703] device ovn-k8s-mp0 entered promiscuous mode
[   74.566114] device br-int entered promiscuous mode
[   74.579314] device ovn-k8s-gw0 entered promiscuous mode
[   74.587375] device br-local entered promiscuous mode
[   75.239718] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   75.302103] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   75.337210] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   75.480347] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   75.745303] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   76.152961] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   76.190249] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   76.833121] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   76.857889] IPv6: ADDRCONF(NETDEV_UP): vetha6309245: link is not ready
[   76.864701] IPv6: ADDRCONF(NETDEV_CHANGE): vetha6309245: link becomes ready
[   76.871962] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   76.901384] 1527e078d8662b6: renamed from vetha6309245
[   76.928056] device 1527e078d8662b6 entered promiscuous mode
[   77.149325] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   77.161405] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   77.177454] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

cynepco3hahue commented 4 years ago

@alosadagrande Thanks Alberto, can you try to write manually 8 to the /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages, check if it succeeded and if not publish the end of dmesg -T. I still do not see any clue why it happened :(

alosadagrande commented 4 years ago

After setting manually "8" on each numa node:

sh-4.4# find /sys -name '*meminfo*' | xargs grep -i huge
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total:     8
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free:      8
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp:      0
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total:     8
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free:      8
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp:      0

sh-4.4#  cat /proc/meminfo  | grep -i huge
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        16777216 kB

Did not require a reboot @cynepco3hahue ?

cynepco3hahue commented 4 years ago

If it succeeded it means that the memory not too much defragmented, so no need to reboot. And if you will reboot will it revert values to 5 and 3?

alosadagrande commented 4 years ago

After rebooting the node did not revert to the previous values but an unexpected ones:

sh-4.4# find /sys -name '*meminfo*' | xargs grep -i huge
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total:     4
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free:      4
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp:      0
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total:     5
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free:      5
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp:      0

Same happens in the other worker node in the same cluster. Well, actually this case the numbers are in the other way around wrt numa nodes:

sh-4.4# find /sys -name '*meminfo*' | xargs grep -i huge
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Total:     5
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Free:      5
/sys/devices/system/node/node0/meminfo:Node 0 HugePages_Surp:      0
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Total:     4
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Free:      4
/sys/devices/system/node/node1/meminfo:Node 1 HugePages_Surp:      0

cynepco3hahue commented 4 years ago

I still do not know what the reson why it failed to allocate hugepages from the first time, but changing the script to

#!/usr/bin/env bash

set -euo pipefail

nodes_path="/sys/devices/system/node"
hugepages_file="${nodes_path}/node${NUMA_NODE}/hugepages/hugepages-${HUGEPAGES_SIZE}kB/nr_hugepages"

if [ ! -f  ${hugepages_file} ]; then
    echo "ERROR: ${hugepages_file} does not exist"
    exit 1
fi

timeout=60
sample=1
current_time=0
while [ $(cat ${hugepages_file}) -ne ${HUGEPAGES_COUNT} ]; do
    echo ${HUGEPAGES_COUNT} > ${hugepages_file}

    current_time=$((current_time + sample))
    if [ $current_time -gt $timeout ]; then
        echo "ERROR: ${hugepages_file} does not have the expected number of hugepages ${HUGEPAGES_COUNT}"
        exit 1
    fi

    sleep $sample
done

helped to solve the problem.

@MarSik @fromanirh Do you have any ideas why it fail to allocate hugepages from the first try? And does it make sense to change the script? My concerns that we do not really have any confirmation that running the while loop for 60 seconds will be enough or not enough.

MarSik commented 4 years ago

Uh.. well a minute should be ok. But I think we should ask someone from the kernel team to find out why this is happening.

ffromani commented 4 years ago

I still do not know what the reson why it failed to allocate hugepages from the first time, but changing the script to
#!/usr/bin/env bash

set -euo pipefail

nodes_path="/sys/devices/system/node"
hugepages_file="${nodes_path}/node${NUMA_NODE}/hugepages/hugepages-${HUGEPAGES_SIZE}kB/nr_hugepages"

if [ ! -f  ${hugepages_file} ]; then
    echo "ERROR: ${hugepages_file} does not exist"
    exit 1
fi

timeout=60
sample=1
current_time=0
while [ $(cat ${hugepages_file}) -ne ${HUGEPAGES_COUNT} ]; do
    echo ${HUGEPAGES_COUNT} > ${hugepages_file}

    current_time=$((current_time + sample))
    if [ $current_time -gt $timeout ]; then
        echo "ERROR: ${hugepages_file} does not have the expected number of hugepages ${HUGEPAGES_COUNT}"
        exit 1
    fi

    sleep $sample
done
helped to solve the problem.

@MarSik @fromanirh Do you have any ideas why it fail to allocate hugepages from the first try? And does it make sense to change the script? My concerns that we do not really have any confirmation that running the while loop for 60 seconds will be enough or not enough.

Retrying is indeed suboptimal but works as mitigation until we figure out the root cause. And can hardly do any harm anyway. Unfortunately no clue about why allocation fails. I think that the earlier in the worker lifecycle we try to allocate, the higher should the success rate (less fragmented memory because less processes run).

I wonder if technologies like KSM may help keep the memory fragmentation at bay if is indeed a memory fragmentation issue.

openshift-kni / performance-addon-operators

Provide possibility to configure uneven split of hugepages between NUMA nodes #279