search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.96k stars 4.65k forks source link

Update kops ami for cve issues - meltdown and spectre #4188

Closed chrislovecnm closed 4 years ago

chrislovecnm commented 6 years ago

Patch kernel in our Jessie and Stretch amis for

(CVE-2017-5715, CVE-2017-5753, and CVE-2017-5754)

poidag-zz commented 6 years ago

Patches required for AMIs to address

https://meltdownattack.com/

CVE-2017-5753 and CVE-2017-5715 refer to Spectre CVE-2017-5754 refers to Meltdown

Meltdown and Spectre exploit critical vulnerabilities in modern processors. These hardware bugs allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs.

chrislovecnm commented 6 years ago

What kernel versions are patched?

chrislovecnm commented 6 years ago

We determine that Linus just updated his branch, and the patch for 4.4 is not available yet.

More info about the exploits https://aws.amazon.com/security/security-bulletins/AWS-2018-013/

chrislovecnm commented 6 years ago

For CoreOS users from @philips https://lkml.org/lkml/2018/1/3/105

chrislovecnm commented 6 years ago

For google container os follow https://support.google.com/faqs/answer/7622138

chrislovecnm commented 6 years ago

/area security

chrislovecnm commented 6 years ago

This has not been merged yet https://lkml.org/lkml/2018/1/3/770 ... once the kernel gurus finish duking it out, we may have the patches in. Not sure when they plan on cutting next kernel release though.

BradErz commented 6 years ago

any progress?

thomaspeitz commented 6 years ago

https://security-tracker.debian.org/tracker/CVE-2017-5753 https://security-tracker.debian.org/tracker/CVE-2017-5715 https://security-tracker.debian.org/tracker/CVE-2017-5754

Jessie currently has no patches, only one stretch cve fixed.

(stretch is the one which will be used with kops 1.9 release and is not released stable yet)

So we still have to wait @BradErz

BobbyJohansen commented 6 years ago

Are you guys planning on updating the Jessie AMI?

chrislovecnm commented 6 years ago

Kops alpha channel has the new AMIs. Please test!!

chrislovecnm commented 6 years ago

PR for the advisories is in https://github.com/kubernetes/kops/pull/4211. Please comment, and review!!!!!

krogon-dp commented 6 years ago

@chrislovecnm This advisory includes the intial work to resolve CVE-2017-5753 and CVE-2017-5754. Debian does not fix CVE-2017-5753 yet. I run spectre-meltdown-checker and it shows that AMI kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2018-01-05 is vulnerable.

Checking vulnerabilities against Linux 4.4.110-k8s #1 SMP Fri Jan 5 16:59:17 UTC 2018 x86_64

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Kernel compiled with LFENCE opcode inserted at the proper places:  NO  (only 38 opcodes found, should be >= 70)
> STATUS:  VULNERABLE
RickyCook commented 6 years ago

this is incorrect; see below

AFAIK one of the Spectre variants can't be fixed through a kernel patch?

So I think this is correct, expected, and as much as we can do. The Spectre variant 1 can only be fixed in hardware I think

krogon-dp commented 6 years ago

Nope. Spectre Variant 2 is corrected with CPU microcode, along with kernel which enables new microcode. Variant 1 & 3 are pure kernel patches.

What's wrong is stating that the advisory fixes Variant 1 (Spectre) & 3 (Meltdown) while only last is correct.

RickyCook commented 6 years ago

ill defer to your much better understanding :)

i hate the mixed messages everyone has about this. that’s possibly the worst part; how hard it is to find reliable summaries

i’ll edit my comment so it’s obvious

chrislovecnm commented 6 years ago

@RickyCook give our advisory a read, and @krogon-dp if you want to add a section to the advisory about the 2nd variant, that would be awesome.

Here is the kops advisory: https://github.com/kubernetes/kops/blob/master/docs/advisories/spectre-meltdown-kernel-update.md

I think the patch is on its way for variant two.

Gentoo always provides super low detailed info about stuff like this https://wiki.gentoo.org/wiki/Project:Security/Vulnerabilities/Meltdown_and_Spectre#sys-kernel.2Fgentoo-sources

Also, this came across my twitter feed about the second variant of spectre https://lists.gnu.org/archive/html/qemu-devel/2018-01/msg01556.html

chrislovecnm commented 6 years ago

@krogon-dp thanks for the info, was 5753 not resolved yet?

philstrong commented 6 years ago

@chrislovecnm fantastic share thanks!

Exceptionally well done advisory

krogon-dp commented 6 years ago

@chrislovecnm about 5753 not yet in ubuntu/debian kernels https://github.com/hannob/meltdownspectre-patches

chrislovecnm commented 6 years ago

@krogon-dp we do not use the distribution kernel. Is the fix in the actual kernel itself?

bcorijn commented 6 years ago

@justinsb I see kernel 4.4.111 has been released with more Spectre related fixes. Can another kops image be built? Thanks for all the work!

TGippert commented 6 years ago

Currently I have seen, there are new AMI's available: kope.io/k8s-1.x-debian-jessie-amd64-hvm-ebs-2018-01-14

Here the results for k8s-1.7-debian-jessie-amd64-hvm-ebs-2018-01-14:

Checking for vulnerabilities against running kernel Linux 4.4.111-k8s #1 SMP Sun Jan 14 19:32:08 UTC 2018 x86_64 CPU is Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'

philstrong commented 6 years ago

So still VULNERABLE for 2 CVEs. Any word on patches for those?

TGippert commented 6 years ago

For those of you speaking german: Very interesting article regarding linux kernel patches and explaining our "new friends" spectre and meltdown and how to protect against them https://www.heise.de/ct/artikel/Die-Neuerungen-von-Linux-4-15-3900646.html

krogon-dp commented 6 years ago

@philstrong it depends on upstream distro patches, which are not available right now: https://github.com/hannob/meltdownspectre-patches#linux-distributions

snoby commented 6 years ago

So I was just looking at the stable channel and it looks like the 01-14 build is being used now: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-01-14

https://github.com/kubernetes/kops/blob/2a5e13660d59184f6269c406a95ff49181408c8d/channels/stable#L24

I loaded it up on a test cluster and ran the spectre-meltdown-checker:

Spectre and Meltdown mitigation detection tool v0.32

Checking for vulnerabilities against running kernel Linux 4.4.111-k8s #1 SMP Sun Jan 14 19:32:08 UTC 2018 x86_64
CPU is  Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Checking count of LFENCE opcodes in kernel:  NO
> STATUS:  VULNERABLE  (only 25 opcodes found, should be >= 70, heuristic to be improved when official patches become available)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
*   Hardware (CPU microcode) support for mitigation
*     The SPEC_CTRL MSR is available:  YES
*     The SPEC_CTRL CPUID feature bit is set:  NO
*   Kernel support for IBRS:  NO
*   IBRS enabled for Kernel space:  NO
*   IBRS enabled for User space:  NO
* Mitigation 2
*   Kernel compiled with retpoline option:  NO
*   Kernel compiled with a retpoline-aware compiler:  NO
> STATUS:  VULNERABLE  (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Checking if we're running under Xen PV (64 bits):  NO
> STATUS:  NOT VULNERABLE  (PTI mitigates the vulnerability)

seems like no change from the 01-05 image.

krogon-dp commented 6 years ago

@snoby See my previous response - there are no upstream patches for debian/ubuntu yet. Moreover there is a known bug [1] in recent Intel microcode (for Variant 2), which results in reverting it [2].

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886998 [2] https://launchpad.net/ubuntu/+source/intel-microcode/3.20180108.1+really20171117.1

krogon-dp commented 6 years ago

^up: seems like 4.4.0-111.134 [1] was released yesterday and addresses this issues [2].

[1] https://launchpad.net/ubuntu/+source/linux-lts-xenial/4.4.0-111.134~14.04.1 [2] https://usn.ubuntu.com/usn/usn-3540-2/

virusdave commented 6 years ago

Any updates on getting kops images with the latest upstream security fixes?

EDIT: Checking in again. Any progress here? I know @justinsb made the last spectre/meltdown image update commits.

Nitecon commented 6 years ago

@chrislovecnm can you comment on images going out now that upstream is available? It's been out for a while and we have some critical stuff in the path so just like others any update would be useful to provide back.

virusdave commented 6 years ago

We're over 2 weeks now since upstream security patches have been available. Is there any progress on this? I'll be very sad if we have to migrate away from kops because images don't get regularly updated with security fixes :(

chrislovecnm commented 6 years ago

@virusdave I am uncertain of the timeline for the update, and I do understand your concern. Most of us have day jobs and volunteer our time for kops. Another option is to use your own AMI. You do not have to use our image.

justinsb commented 6 years ago

I'll build and push a new image into the alpha channel, or you can just use the latest debian/ubuntu/centos/rhel AMI (with kops) if you'd prefer :-)

justinsb commented 6 years ago

Docs on how to use an arbitrary AMI are here: https://github.com/kubernetes/kops/blob/master/docs/images.md

Nitecon commented 6 years ago

@justinsb is the only change the ixgbevf and ena drivers because it looks like this was addressed in upstream on andsens/bootstrap-vz. If there is nothing off the top of your head then I'm probably gonna try build it off master to see how it works and will respond with outcome here if it works

justinsb commented 6 years ago

So I have built the latest 4.4 LTS kernel with a newer gcc, which now passes spectre v2 (meltdown is already addressed).

Spectre v1 looks to be unfixed in Debian: https://security-tracker.debian.org/tracker/CVE-2017-5753 and I believe is still unfixed in the stable kernels. If someone knows otherwise, please let me know what I'm missing :-)

@Nitecon you can see the changes here: https://github.com/kubernetes/kube-deploy/blob/master/imagebuilder/templates/1.8-stretch.yml But if you want to use the official Stretch image, that should be easier, or the official Ubuntu image, that'll be even easier. It looks like Debian hasn't yet patched Specter v1, but I believe Ubuntu has applied their own fix for Spectre v1.

But also @Nitecon what is the fix you see upstream in andsens/bootstrap-vz ? If you can point me to the commit, I can be sure we have it incorporated.

I guess it's still better to do a new image with the mitigation for Spectre v2, even without mitigation for Spectre v1. Unless I hear otherwise I'll push that image into the alpha channel and proceed with that.

Nitecon commented 6 years ago

@justinsb i was referring to the updates that you provided in your fork has been implemented in andsens/bootstrap-vz master so the custom fork shouldn't be required anymore. Beyond that I was still reading through the imagebuilder setup and it looks like it's basically 100% locked to debian at this point and all additional distros would require a lot of dev work, so with that said I'm probably gonna work on a debian based image first to mimic official.

I'll be building all this with hashicorps packer. If that works I may add ubuntu && centos builders also if people are interested. From what I can see so far it looks like it's only a customized kernel to add cgroups and docker that needs to be installed along with awscli? If there is anything else please let me know where to look for them. I'll do what I can to get this working and will be happy to provide it as OSS for anyone to consume if there is interest.

justinsb commented 6 years ago

@Nitecon the kubernetes AMI that we build is the 4.4 LTS kernel (to avoid some kernel panics with the debian jessie kernel, primarily); some tweaks to better run docker (e.g. more inodes), preinstallation of docker for speed, and then we preinstall some utilities mostly for convenience.

I'm confused about what you're trying to achieve though - can you just use the official Ubuntu / Debian / RHEL / CentOS image that has the kernel you want? Or are you going to build your own kernel? And if so, is it going to mitigate Spectre v1 :stuck_out_tongue: ?

Nitecon commented 6 years ago

The whole idea is to mitigate Spectre and make future paching easier while not loosing custom compiled kernels as required by kops/kubernetes. Just using the official images would still reduce spin up time / deployment time and I'm trying to avoid going slower by just pre-building it. Would make life easier to test across multiple distros too though.

justinsb commented 6 years ago

New image in alpha channel, includes 4.4.115 compiled with gcc 7.3. Mitigates spectre v2. Spectre v1 not yet addressed in the 4.4 kernel.

https://github.com/kubernetes/kops/pull/4411#issuecomment-364305807

virusdave commented 6 years ago

Thanks for all the updates. Sorry if I came off brusquely, and thanks for the link for custom images in kops. We might use that temporarily while the official channels get updated.

Keep up the great work all!

prateeknayak commented 6 years ago

Hi, any updates on latest alpha channel image and when this image will be available in the stable channel for spectre v2 and v1 fixes ?

thomasjungblut commented 6 years ago

has anyone been using the new alpha channel image already and can give some insight of how it went?

23doors commented 6 years ago

Using 1.9.3 on k8s-1.8-debian-stretch-amd64-hvm-ebs-2018-02-08 and everything seems to be working fine. But when it comes to vulnerability - seems there are no changes.

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Kernel has array_index_mask_nospec:  NO
* Kernel has the Red Hat/Ubuntu patch:  NO
* Checking count of LFENCE instructions following a jump in kernel...  NO  (only 3 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS:  VULNERABLE  (Kernel source needs to be patched to mitigate the vulnerability)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigation 1
  * Kernel is compiled with IBRS/IBPB support:  NO
  * Currently enabled features
    * IBRS enabled for Kernel space:  NO
    * IBRS enabled for User space:  NO
    * IBPB enabled:  NO
* Mitigation 2
  * Kernel compiled with retpoline option:  NO
  * Kernel compiled with a retpoline-aware compiler:  NO
> STATUS:  VULNERABLE  (IBRS hardware + kernel support OR kernel with retpoline are needed to mitigate the vulnerability)

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Kernel supports Page Table Isolation (PTI):  YES
* PTI enabled and active:  YES
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (PTI mitigates the vulnerability)
kdillane commented 6 years ago

Any updates on the status here? Is there an expected timeline for including the updated AMIs with kops?

jjo commented 6 years ago

Related: kubernetes/kops#5131 for the so called "variant-4".

adityats2020 commented 6 years ago

Stretch has been patched by Debian https://security-tracker.debian.org/tracker/CVE-2017-5753 https://security-tracker.debian.org/tracker/CVE-2017-5715

idealhack commented 6 years ago

Any updates on this?

bcorijn commented 6 years ago

Just a quick update on the results of https://github.com/speed47/spectre-meltdown-checker on the latest AMI kope.io/k8s-1.10-debian-stretch-amd64-hvm-ebs-2018-08-17, in case people are curious

CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec: perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
 YES  (1 occurrence(s) found of x86 64 bits array_index_mask_nospec())
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  NO
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  NO
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  NO
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
  * Kernel supports RSB filling:  YES
> STATUS:  NOT VULNERABLE  (Full retpoline is mitigating the vulnerability)
IBPB is considered as a good addition to retpoline for Variant 2 mitigation, but your CPU microcode doesn't support it

CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 [rogue system register read] aka 'Variant 3a'
* CPU microcode mitigates the vulnerability:  NO
> STATUS:  VULNERABLE  (an up-to-date CPU microcode is needed to mitigate this vulnerability)

CVE-2018-3639 [speculative store bypass] aka 'Variant 4'
* Mitigated according to the /sys interface:  NO  (Vulnerable)
* Kernel supports speculation store bypass:  YES  (found in /proc/self/status)
> STATUS:  VULNERABLE  (Your CPU doesn't support SSBD)

CVE-2018-3615/3620/3646 [L1 terminal fault] aka 'Foreshadow & Foreshadow-NG'
> STATUS:  VULNERABLE  (your CPU is known to be vulnerable, and your kernel doesn't report that it mitigates the issue, but more thorough mitigation checking by this script is being worked on (check often for new versions!))