Seccomp

[pweil-]

Description

Seccomp support is providing the ability to define seccomp profiles and configure pods to run with those profiles. This includes the ability control usage of the profiles via PSP as well as maintaining the ability to run as unconfined or with the default container runtime profile.

KEP: sig-node/20190717-seccomp-ga.md Latest PR to update the KEP: #1747

Progress Tracker

• [x] Alpha
  • [x] Write and maintain draft quality doc: available in downstream OpenShift https://github.com/openshift/openshift-docs/pull/2975
  • [x] Design Approval
  • [x] Design Proposal #24602
  • [x] Decide which repo this feature's code will be checked into. Not everything needs to land in the core kubernetes repo. REPO-NAME
  • [x] Initial API review (if API). Maybe same PR as design doc. https://github.com/kubernetes/kubernetes/pull/24602
    • Any code that changes an API (/pkg/apis/...)
    • cc @kubernetes/api
  • [x] Identify shepherd (your SIG lead and/or kubernetes-pm@googlegroups.com will be able to help you). My Shepherd is: replace.me@replaceme.com (and/or GH Handle)
    • A shepherd is an individual who will help acquaint you with the process of getting your feature into the repo, identify reviewers and provide feedback on the feature. They are not (necessarily) the code reviewer of the feature, or tech lead for the area.
    • The shepherd is not responsible for showing up to Kubernetes-PM meetings and/or communicating if the feature is on-track to make the release goals. That is still your responsibility.
  • [x] Identify secondary/backup contact point. My Secondary Contact Point is: replace.me@replaceme.com (and/or GH Handle)
  • [x] Write (code + tests + docs) then get them merged. https://github.com/kubernetes/kubernetes/pull/25324 https://github.com/kubernetes/kubernetes/pull/26710 https://github.com/kubernetes/kubernetes/pull/27036
  • [x] Code needs to be disabled by default. Verified by code OWNERS
  • [x] Minimal testing: limited e2e tests https://github.com/kubernetes/kubernetes/blob/33ebe1f18b9cf5909931376f656e19e80ac9ac1c/test/e2e/security_context.go#L110
  • [x] Minimal docs
    • cc @kubernetes/docs on docs PR
    • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
    • New apis: Glossary Section Item in the docs repo: kubernetes/kubernetes.github.io
  • [x] Update release notes https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md/#changes-since-v130-alpha4
• [x] Beta
  • [x] Testing is sufficient for beta
  • [x] User docs with tutorials
  • Updated walkthrough / tutorial in the docs repo: kubernetes/kubernetes.github.io
  • cc @kubernetes/docs on docs PR
  • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
  • [x] Thorough API review
  • cc @kubernetes/api
• [x] Stable
  • [x] docs/proposals/foo.md moved to docs/design/foo.md
  • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off
  • [x] Soak, load testing
  • [x] detailed user docs and examples
  • cc @kubernetes/docs
  • cc @kubernetes/feature-reviewers on this issue to get approval before checking this off

_FEATURESTATUS is used for feature tracking and to be updated by @kubernetes/feature-reviewers. FEATURE_STATUS: IN_DEVELOPMENT

More advice:

Design

• Once you get LGTM from a @kubernetes/feature-reviewers member, you can check this checkbox, and the reviewer will apply the "design-complete" label.

Coding

• Use as many PRs as you need. Write tests in the same or different PRs, as is convenient for you.
• As each PR is merged, add a comment to this issue referencing the PRs. Code goes in the http://github.com/kubernetes/kubernetes repository, and sometimes http://github.com/kubernetes/contrib, or other repos.
• When you are done with the code, apply the "code-complete" label.
• When the feature has user docs, please add a comment mentioning @kubernetes/feature-reviewers and they will check that the code matches the proposed feature and design, and that everything is done, and that there is adequate testing. They won't do detailed code review: that already happened when your PRs were reviewed. When that is done, you can check this box and the reviewer will apply the "code-complete" label.

Docs

• [x] Write user docs and get them merged in.
• User docs go into http://github.com/kubernetes/kubernetes.github.io.
• When the feature has user docs, please add a comment mentioning @kubernetes/docs.
• When you get LGTM, you can check this checkbox, and the reviewer will apply the "docs-complete" label.

[pweil-]

@derekwaynecarr @sttts @erictune didn't see an issue for this but it is already in alpha. Creating this as the reminder to push it through to beta and stable.

@sttts could you provide the appropriate links to docs and PRs? I think you are closest to this code.

[derekwaynecarr]

@pweil- @sttts - per our discussion, this is a feature we would like to sponsor in Kubernetes 1.6 under @kubernetes/sig-node

[idvoretskyi]

@pweil- @derekwaynecarr please, confirm that this feature has to be set with 1.6 milestone.

[sttts]

@idvoretskyi we target to move it to beta for 1.6.

[idvoretskyi]

@sttts thanks.

[bgrant0607]

Looks like this is still alpha:

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/seccomp.md https://github.com/kubernetes/kubernetes/blob/master/pkg/api/annotation_key_constants.go#L35

And I couldn't find any documentation on kubernetes.io/docs.

[idvoretskyi]

@pweil- any updates for 1.8? Is this feature still on track for the release?

[pweil-]

@idvoretskyi this was not a priority for 1.8. @php-coder can you add a card to this for our PM planning? We need to stop letting this fall through the cracks and get it moved to beta and GA.

[idvoretskyi]

@pweil- if this feature is not planned for 1.8 - please, update the milestone with the "next-milestone" or "1.9"

[tallclair]

I'd like to see this get to beta. Priorities (or requirements) for that include:

1. Annotations (Pod & PodSecurityPolicy) must be moved to fields on the container SecurityContext (see https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md#alpha-field-in-existing-api-version)
2. Settle on the OCI spec seccomp format, and define a Kubernetes default profile (can we reuse Docker's?) https://github.com/kubernetes/kubernetes/issues/39128 a. Figure out how the Kubernetes profile is delivered to the container runtime via CRI (/cc @yujuhong @Random-Liu ) b. docker/default should still be allowed if the runtime is docker (for backwards compatibility)
3. The Kubernetes default profile should be the new default. For backwards compatibility, this MUST be optional behavior (i.e. flag controlled). https://github.com/kubernetes/kubernetes/issues/39845

Is anyone interested in driving this work for the 1.9 (or 1.10) milestone? @jessfraz @kubernetes/sig-auth-feature-requests and @kubernetes/sig-node-feature-requests I'm looking at you :wink:

Also relevant: https://github.com/kubernetes/community/pull/660 (do we need to settle the decisions in that PR before proceeding?)

[tallclair]

/cc @destijl

[jessfraz]

If someone has time and wants to do it they are more than welcome to and I will help answer any questions

On Sep 22, 2017 20:54, "Tim Allclair (St. Clair)" notifications@github.com wrote:

/cc @destijl https://github.com/destijl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/135#issuecomment-331593048, or mute the thread https://github.com/notifications/unsubscribe-auth/ABYNbDldlrwbOP75Y2AKM-bUFLnwrq0eks5slFbcgaJpZM4KgBy_ .

[jessfraz]

ok I will update the proposal and start on this tomorrow if no one else will ;)

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

[jlk]

Hey @jessfraz not sure if you got anywhere on this - I don't have bandwidth to code it, but happy to help test...

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

[php-coder]

/reopen /lifecycle frozen /remove-lifecycle rotten

[k8s-ci-robot]

@php-coder: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to [this](https://github.com/kubernetes/features/issues/135#issuecomment-376129291): >/reopen >/lifecycle frozen >/remove-lifecycle rotten > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/devel/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[smarterclayton]

/reopen /lifecycle frozen

On Mon, Mar 26, 2018 at 7:07 AM, k8s-ci-robot notifications@github.com wrote:

@php-coder https://github.com/php-coder: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this https://github.com/kubernetes/features/issues/135#issuecomment-376129291 :

/reopen /lifecycle frozen /remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here https://git.k8s.io/community/contributors/devel/pull-requests.md. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue: repository.

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/135#issuecomment-376129294, or mute the thread https://github.com/notifications/unsubscribe-auth/ABG_p9EwKebniej_GySRKSvzrCMITOA1ks5tiMvrgaJpZM4KgBy_ .

[k8s-ci-robot]

@smarterclayton: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to [this](https://github.com/kubernetes/features/issues/135#issuecomment-376395185): >/reopen >/lifecycle frozen > >On Mon, Mar 26, 2018 at 7:07 AM, k8s-ci-robot >wrote: > >> @php-coder : you can't re-open an issue/PR >> unless you authored it or you are assigned to it. >> >> In response to this >> >> : >> >> /reopen >> /lifecycle frozen >> /remove-lifecycle rotten >> >> Instructions for interacting with me using PR comments are available here >> . If >> you have questions or suggestions related to my behavior, please file an >> issue against the kubernetes/test-infra >> >> repository. >> >> — >> You are receiving this because you are on a team that was mentioned. >> Reply to this email directly, view it on GitHub >> , >> or mute the thread >> >> . >> > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/devel/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[idvoretskyi]

/reopen

[k8s-ci-robot]

@idvoretskyi: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to [this](https://github.com/kubernetes/features/issues/135#issuecomment-376399770): >/reopen > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/devel/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[smarterclayton]

Ihor 1, bot 0

[justaugustus]

@pweil- @php-coder @jessfraz Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  • stage/{alpha,beta,stable}
  • sig/*
  • kind/feature

cc @idvoretskyi

[tallclair]

@wangzhen127 is working on it, but can't be assigned as he's not a member yet.

https://github.com/kubernetes/kubernetes/pull/62662 https://github.com/kubernetes/kubernetes/pull/62671

[justaugustus]

Thanks for the update, Tim! /remove-lifecycle frozen

[justaugustus]

@pweil- @tallclair -- We're doing one more sweep of the 1.11 Features tracking spreadsheet. Would you mind filling in any incomplete / blank fields for this feature's line item?

[justaugustus]

@pweil- @tallclair -- this feature has been removed from the 1.11 milestone, as there have been no updates w.r.t. progress or docs.

cc: @jberkus

[justaugustus]

@pweil- @tallclair @kubernetes/sig-auth-feature-requests @kubernetes/sig-node-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  • Alpha release target (x.y)
  • Beta release target (x.y)
  • Stable release target (x.y)

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[tallclair]

No changes planned for 1.12

[justaugustus]

Thanks for the update, @tallclair!

[stlaz]

Is anyone interested in driving this work for the 1.9 (or 1.10) milestone? @jessfraz @kubernetes/sig-auth-feature-requests and @kubernetes/sig-node-feature-requests I'm looking at you wink

@tallclair I can try to pick this up now if still desirable

[k8s-ci-robot]

@stlaz: Reiterating the mentions to trigger a notification: @kubernetes/sig-auth-feature-requests, @kubernetes/sig-node-feature-requests

In response to [this](https://github.com/kubernetes/features/issues/135#issuecomment-415009299): >> Is anyone interested in driving this work for the 1.9 (or 1.10) milestone? @jessfraz @kubernetes/sig-auth-feature-requests and @kubernetes/sig-node-feature-requests I'm looking at you wink > >@tallclair I can try to pick this up now if still desirable Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[wangzhen127]

@stlaz, this feature is still desired. I have spent some time adding seccomp profiles to addons as the first steps of #39845. But I haven't got enough time to push the feature. It would be nice if you like working on this. Any help is welcome! :)

[stlaz]

@wangzhen127 Thanks, I was trying to go through the things that were done and the issues opened related to this feature. It would seem that the comment https://github.com/kubernetes/features/issues/135#issuecomment-331592961 still holds and summarizes exactly the work that needs to be done now.

I also noticed you were trying to add a FeatureGate for this, but closed the PR, why was that?

P.S.: Sorry for the late response, I was away for a bit.

[wangzhen127]

It would seem that the comment #135 (comment) still holds and summarizes exactly the work that needs to be done now.

That is right. One more thing I would like to add is to have a "complaint" mode. So users have a choice of getting warnings (in logs) of using forbidden syscalls rather than kill. Logging seccomp actions are available with Linux kernel 4.14+ (seccomp doc). It is possible that older kernel versions are still being used. So we need to handle that. This will also need to be added to OCI spec.

I also noticed you were trying to add a FeatureGate for this, but closed the PR, why was that?

The purpose of that feature gate was to change seccomp default profile from unconfined to "runtime/default". I got many concerns on backward compatibility, so that seemed unlikely to happen. We are currently lack of a plan of changing security defaults in general because those are breaking. The best approach I currently think is to push seccomp to stable and it still has to be an opt-in feature rather than opt-out.

[stlaz]

Logging seccomp actions are available with Linux kernel 4.14+ (seccomp doc).

I guess since we'll be defining a Kubernetes-specific default seccomp format as a part of the second step, we could also have one that would log instead. Is there enough value for it? Could it be used for people to transition from "unconfined" to "kube/default" when the latter fails too much? Would they care for it to switch-to switch-back? There are LTS distributions using 4.13- Linux kernels (Debian - 8,9; RHEL - 6, 7; Ubuntu LTS - 14.04, 16.04), so Kernel compatibility is definitely something to keep in mind.

I got many concerns on backward compatibility, so that seemed unlikely to happen.

The container runtimes had to go through this change in the past, too, when they were enabling seccomp for the first time. Right now at least docker ships with default behavior which is less permitting than "unconfined", which could have possibly broken someone. I don't think we're doing anything wrong if we just follow the behavior of the underlying components (that also provide the choice of turning this behavior off).

[wangzhen127]

Is there enough value for it?

This can be discussed. My original thought was to change default from unconfined to logging. So we do not have backward compatibility issue. And if we can somehow collect data and show that in X% of cases, we see nothing logged, meaning the default profile would not break things. Then we can propose to change logging to kill. Collecting data part is tricky and can be a lot of work. Even if we don't actually go that route, I think having a logging profile would benefit people when they want to try seccomp out but are not confident yet.

The container runtimes had to go through this change in the past, too, when they were enabling seccomp for the first time. Right now at least docker ships with default behavior which is less permitting than "unconfined", which could have possibly broken someone. I don't think we're doing anything wrong if we just follow the behavior of the underlying components (that also provide the choice of turning this behavior off).

When docker changed the default value, kubernetes explicitly reset such value to unconfined. I reached out to sig-architecture folks offline before and they are very worried about the backward compatibility issue.

[stlaz]

And if we can somehow collect data and show that in X% of cases, we see nothing logged, meaning the default profile would not break things.

I like this idea. The hard part is of course getting the data, I have no idea how to pull that one off. Also, we'd have to first propose this change to the OCI spec and then probably implement it for at least one container runtime, right? Would that be OK to happen in the Beta part of the lifecycle?

When docker changed the default value, kubernetes explicitly reset such value to unconfined. I reached out to sig-architecture folks offline before and they are very worried about the backward compatibility issue.

I see. Perhaps we could indeed just go with the "unconfined" profile as the default one (possibly replacing it with something like kube/logging later on). It seems that this might then rather be controlled by PSPs in a deny-rule manner, where we start with the assumption that everything is allowed by default and we're only cutting the privileges further on. Having a flag control over this may be useful for cases where PSPs are turned off, though, so that one should go in too, yet having these two mechanisms used at once would probably be a bit messy.

I guess it's a bit different direction than originally considered - it goes against the work done in https://github.com/kubernetes/kubernetes/issues/39845, but if we agree on the above, we should then think of the seccomp profiles we'll eventually ship. So far I'm seeing runtime/default, kube/default, kube/logging, along with the option to set the profile to unconfined. The rest is of course the ability to have localhost/.* profiles, which is already provided by the current implementation.

[wangzhen127]

Would that be OK to happen in the Beta part of the lifecycle?

Sounds good to me. Though I think it helps to start the OCI spec proposal early.

Go with "unconfined" as the default for now sounds good to me. For kubernetes/kubernetes#39845, I added annotations to addons. And I don't think we can go any further.

So far I'm seeing runtime/default, kube/default, kube/logging, along with the option to set the profile to unconfined. The rest is of course the ability to have localhost/.* profiles, which is already provided by the current implementation.

Works for me. For kube/default, we may just start with docker/default.

[jlk]

Logging seccomp actions are available with Linux kernel 4.14+ (seccomp doc).

My understanding is this logs the action with the PID - not necessarily container-related info. So either auditd or some other daemon on the host will need to do a lookup/mapping for the log to be really useful...

[tallclair]

And if we can somehow collect data and show that in X% of cases, we see nothing logged, meaning the default profile would not break things. Then we can propose to change logging to kill. Collecting data part is tricky and can be a lot of work.

Didn't @jessfraz already do that when launching the docker default profile for docker? If that isn't a strong enough signal, it's going to be very difficult to collect a stronger one...

[stlaz]

@tallclair you're right, I got kind of lost in all the issues' comments. For the reference, here's the comment stating Dockerfiles were checked: https://github.com/kubernetes/community/pull/660#issuecomment-303860107. I guess we could be safe having a "killing" default after all.

[kacole2]

Hi This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen. Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[stlaz]

/remove-lifecycle rotten

[kacole2]

Hello @stlaz @pweil- , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP to be included into 1.15

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.