XCP-ng 7.6 Dom0 Vulnerable to multiple spectre and meltdown

[sammcj]

I noticed thatXCP-ng 7.6 (dom0) with the latest updates installed is still very vulnerable to several serious CVEs:

# ./spectre-meltdown-checker.sh  --explain
Spectre and Meltdown mitigation detection tool v0.40

Checking for vulnerabilities on current system
Kernel is Linux 4.4.0+10 #1 SMP Thu Sep 13 08:55:11 UTC 2018 x86_64
CPU is Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES  (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability:  YES  (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available:  YES
    * CPU indicates L1D flush capability:  NO
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  NO
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO):  NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system:  NO
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA):  NO
  * CPU supports Software Guard Extensions (SGX):  NO
  * CPU microcode is known to cause stability problems:  NO  (model 0x2d family 0x6 stepping 0x7 ucode 0x714 cpuid 0x206d7)
  * CPU microcode is the latest known available version:  YES  (latest version is 0x714 dated 2018/05/08 according to builtin MCExtractor DB v84 - 2018/09/27)
* CPU vulnerability to the speculative execution attack variants
  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass):  YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection):  YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load):  YES
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read):  YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass):  YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault):  NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault):  YES

CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass'
* Kernel has array_index_mask_nospec:  NO
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  NO
* Checking count of LFENCE instructions following a jump in kernel...  NO  (only 4 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS:  VULNERABLE  (Kernel source needs to be patched to mitigate the vulnerability)

> How to fix: Your kernel is too old to have the mitigation for Variant 1, you should upgrade to a newer kernel. If you're using a Linux distro and didn't compile the kernel yourself, you should upgrade your distro to get a newer kernel.

CVE-2017-5715 aka 'Spectre Variant 2, branch target injection'
* Mitigation 1
  * Kernel is compiled with IBRS support:  NO
    * IBRS enabled and active:  UNKNOWN
  * Kernel is compiled with IBPB support:  NO
    * IBPB enabled and active:  NO
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  NO
> STATUS:  VULNERABLE  (IBRS+IBPB or retpoline+IBPB is needed to mitigate the vulnerability)

> How to fix: To mitigate this vulnerability, you need either IBRS + IBPB, both requiring hardware support from your CPU microcode in addition to kernel support, or a kernel compiled with retpoline and IBPB, with retpoline requiring a retpoline-aware compiler (re-run this script with -v to know if your version of gcc is retpoline-aware) and IBPB requiring hardware support from your CPU microcode. The retpoline + IBPB approach is generally preferred as the performance impact is lower. More information about how to enable the missing bits for those two possible mitigations on your system follow. You only need to take one of the two approaches.

> How to fix: Your kernel doesn't have IBPB support, so you need to either upgrade your kernel (if you're using a distro) or recompiling a more recent kernel.

> How to fix: Your kernel doesn't have IBRS support, so you need to either upgrade your kernel (if you're using a distro) or recompiling a more recent kernel.

CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load'
* Kernel supports Page Table Isolation (PTI):  NO
  * PTI enabled and active:  NO
  * Reduced performance impact of PTI:  NO  (PCID/INVPCID not supported, performance impact of PTI will be significant)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Xen Dom0s are safe and do not require PTI)

This host is a Xen Dom0. Please make sure that you are running your DomUs
in HVM, PVHVM or PVH mode to prevent any guest-to-host / host-to-guest attacks.

See https://blog.xenproject.org/2018/01/22/xen-project-spectre-meltdown-faq-jan-22-update/ and XSA-254 for details.

CVE-2018-3640 aka 'Variant 3a, rogue system register read'
* CPU microcode mitigates the vulnerability:  YES
> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)

CVE-2018-3639 aka 'Variant 4, speculative store bypass'
* Kernel supports speculation store bypass:  NO
> STATUS:  VULNERABLE  (your kernel needs to be updated)

> How to fix: You have a recent-enough CPU microcode but your kernel is too old to use the new features exported by your CPU's microcode. If you're using a distro kernel, upgrade your distro to get the latest kernel available. Otherwise, recompile the kernel from recent-enough sources.

CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault'
* CPU microcode mitigates the vulnerability:  N/A
> STATUS:  NOT VULNERABLE  (your CPU vendor reported your CPU model as not vulnerable)

CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault'
* Kernel supports PTE inversion:  NO
* PTE inversion enabled and active:  UNKNOWN  (sysfs interface not available)
> STATUS:  VULNERABLE  (Your kernel doesn't support PTE inversion, update it)

CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault'
* This system is a host running a hypervisor:  YES
* Mitigation 1 (KVM)
  * EPT is disabled:  N/A  (the kvm_intel module is not loaded)
* Mitigation 2
  * L1D flush is supported by kernel:  NO
  * L1D flush enabled:  UNKNOWN  (can't find or read /sys/devices/system/cpu/vulnerabilities/l1tf)
  * Hardware-backed L1D flush supported:  NO  (flush will be done in software, this is slower)
  * Hyper-Threading (SMT) is enabled:  YES
> STATUS:  VULNERABLE  (disable EPT or enabled L1D flushing to mitigate the vulnerability)

> SUMMARY: CVE-2017-5753:KO CVE-2017-5715:KO CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:KO CVE-2018-3615:OK CVE-2018-3620:KO CVE-2018-3646:KO

A false sense of security is worse than no security at all, see --disclaimer

tldr - bad things:

• CVE-2017-5753:KO
• CVE-2017-5715:KO
• CVE-2018-3639:KO
• CVE-2018-3620:KO
• CVE-2018-3646:KO

In fact, the only CVE fixed in XCP-ng 7.6 compared to XenServer 7.2 is CVE-2018-3640.

Tooling used: The very helpful spectre-meltdown-checker

[sammcj]

Note that CentOS 7 VMs, running in HVM mode with the latest stable kernel (4.20.0-1.el7.elrepo.x86_64) is far better off that the dom0:

SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:KO

aka, just CVE-2018-3646

[olivierlambert]

Hi,

FYI, I already asked some Citrix dev about it: this script doesn't check the specific stuff needed to be checked for the dom0, which contains specific patches (remember: dom0 kernel is NOT an upstream kernel, it contains around 300 patches from Citrix). Closing the issue (but not the discussion). We can consider this as a "false alarm".

[sammcj]

Very interesting @olivierlambert and thanks for that information. 👍

I'm a little surprised as looking at the code for spectre-meltdown-checker I can see it's testing actual kernel build options, flags and capabilities - not simply things like the kernel version / number / build date / RHEL (or CentOS) patches etc...

What I might need to do when I get time is prove that one of these CVEs is exploitable and - hopefully as Citrix says - it's not :)

[olivierlambert]

I asked that months ago so I don't remember the technical answer, but it was convincing enough so I decided to not continue on this side.

Now, your next step will be to discover that current XS/XCP-ng uses CentOS 7.2 packages, ie very outdated and containing CVE's fixed in upstream CentOS since. And then you'll write an email to secure at citrix dot com about it, and they'll explain to you that's not a really a problem for no good reasons :wink: Finally, you'll spend days comparing CentOS upstream packages and those in XS/XCP-ng, and starting to update them. You'll enter in a world of pain, even reporting/helping XS project by reporting circular deps, but you'll make it.

[sammcj]

I don’t have an appropriate emoji for this 😂😥😭

-- Sam McLeod

On 7 Jan 2019, at 16:30, Olivier Lambert notifications@github.com wrote:

I asked that months ago so I don't remember the technical answer, but it was convincing enough so I decided to not continue on this side.

Now, your next step will be to discover that current XS/XCP-ng uses CentOS 7.2 packages, ie very outdated and containing CVE's fixed in upstream CentOS since. And then you'll write an email to secure at citrix dot com about it, and they'll explain to you that's not a really a problem for no good reasons 😉 Finally, you'll spend days comparing CentOS upstream packages and those in XS/XCP-ng, and starting to update them. You'll enter in a world of pain, even reporting/helping XS project by reporting circular deps, but you'll make it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[olivierlambert]

Feel free to test our experimental "fully updated" Dom0 (not in production of course). Being more upstream is one of our objective. Citrix told us that we should wait for XS 8.0 to get more recent packages. That doesn't answer CVEs nor even the current LTS status, in terms of security. It seems there is no imminent threats, but as I said to them, it's a bit risky due to a lot of minor CVEs piling up (death by thousand paper-cuts).

Also, it also seems (within XS team) there is no proper workflow (even manual) to try to keep using latest CentOS packages. We'll see on our side what to do with that. We hope to be more close to latest CentOS.

[sammcj]

Would be lovely to see a clean CentOS based XCP based platform, much easier said than done I know and certainly worth waiting for RHEL 8 to come out of beta.

-- Sam McLeod

On 7 Jan 2019, at 20:55, Olivier Lambert notifications@github.com wrote:

Feel free to test our experimental "fully updated" Dom0 (not in production of course). Being more upstream is one of our objective. Citrix told us that we should wait for XS 8.0 to get more recent packages. That doesn't answer CVEs nor even the current LTS status, in terms of security. It seems there is no imminent threats, but as I said to them, it's a bit risky due to a lot of minor CVEs piling up (death by thousand paper-cuts).

Also, it also seems (within XS team) there is no proper workflow (even manual) to try to keep using latest CentOS packages. We'll see on our side what to do with that. We hope to be more close to latest CentOS.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.