search

openshift / cluster-node-tuning-operator

Manage node-level tuning by orchestrating the tuned daemon.
Apache License 2.0
102 stars 105 forks source link

OCPBUGS-31692: [release-4.15] [manual]: Systemd processes not being moved to cpuset/systemd.slice fix #1016

Closed rbaturov closed 7 months ago

rbaturov commented 7 months ago

This is a manual backport for #992

The script cpuset-configure.sh is responsible to move the systemd processes to the cpuset/systemd.slice cgroup and is executed in a form of a service (cpuset-configure.service).

In the current implementation, the script is executed too early - some system processes are yet to be created. This in turn leads to them not being moved to the custom system slice.

Moreover, in the current implementation, the script is executed before the network-online.target. The intention was to execute the script before kubelet and crio services are initialized (by the fact network-online.target is a common parent) in order to make sure that no workload pods are starting before we are making this transition.

The fix I'm proposing consist of the following changes:

  1. Adding an After statements - The script will start once crio service is initialized, due to the fact it's initialized in the very end of the boot process, just a bit before kubelet. Thereby we can ensure late starting processes do not fall between the cracks.
  2. Narrowing down the Before statement to a more accurate one, reflecting its original intention. (Running the script before kubelet only would be enough guarantee no workload pods are started at that time).

When we are using cgroups v1 we are counting on the cpuset-configure.service to move all the system services to the custom system.slice. This test ensures the service indeed moved them.

It is also a good practice to check for similar errors on cgroup v2 systems.

rbaturov commented 7 months ago

/jira cherrypick OCPBUGS-30569

openshift-ci-robot commented 7 months ago

@rbaturov: Jira Issue OCPBUGS-30569 has been cloned as Jira Issue OCPBUGS-31692. Will retitle bug to link to clone. /retitle OCPBUGS-31692: [release-4.15] [manual]: Systemd processes not being moved to cpuset/systemd.slice fix

In response to [this](https://github.com/openshift/cluster-node-tuning-operator/pull/1016#issuecomment-2034435973): >/jira cherrypick OCPBUGS-30569 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-node-tuning-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci-robot commented 7 months ago

@rbaturov: This pull request references Jira Issue OCPBUGS-31692, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/cluster-node-tuning-operator/pull/1016): >This is a manual backport for #992 > >* Systemd processes not being moved to cpuset/systemd.slice fix > >The script cpuset-configure.sh is responsible to move the systemd processes to the cpuset/systemd.slice cgroup and is executed in a form of a service (cpuset-configure.service). > >In the current implementation, the script is executed too early - some system processes are yet to be created. >This in turn leads to them not being moved to the custom system slice. > >Moreover, in the current implementation, the script is executed before the network-online.target. The intention was to execute the script before kubelet and crio services are initialized (by the fact network-online.target is a common parent) in order to make sure that no workload pods are starting before we are making this transition. > >The fix I'm proposing consist of the following changes: >1. Adding an After statements - The script will start once crio service is initialized, due to the fact it's initialized in the very end of the boot process, just a bit before kubelet. >Thereby we can ensure late starting processes do not fall between the cracks. >2. Narrowing down the Before statement to a more accurate one, reflecting its original intention. (Running the script before kubelet only would be enough guarantee no workload pods are started at that time). > > > >* Added a test to verify system processes are in the correct cgroup > >When we are using cgroups v1 we are counting on the cpuset-configure.service to move all the system services to the custom system.slice. This test ensures the service indeed moved them. > >It is also a good practice to check for similar errors on cgroup v2 systems. > > > >--------- Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-node-tuning-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
rbaturov commented 7 months ago

/retest-required

mrniranjan commented 7 months ago

/jira refresh

openshift-ci-robot commented 7 months ago

@mrniranjan: This pull request references Jira Issue OCPBUGS-31692, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.15.z) matches configured target version for branch (4.15.z) * bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST) * dependent bug [Jira Issue OCPBUGS-30569](https://issues.redhat.com//browse/OCPBUGS-30569) is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA)) * dependent [Jira Issue OCPBUGS-30569](https://issues.redhat.com//browse/OCPBUGS-30569) targets the "4.16.0" version, which is one of the valid target versions: 4.16.0 * bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira (mniranja@redhat.com), skipping review request.

In response to [this](https://github.com/openshift/cluster-node-tuning-operator/pull/1016#issuecomment-2039611965): >/jira refresh Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-node-tuning-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
jmencak commented 7 months ago

/approved /label backport-risk-assessed Any reason some of the e2e tests are excluded from the manual backport? /hold

rbaturov commented 7 months ago

/approved

/label backport-risk-assessed

Any reason some of the e2e tests are excluded from the manual backport?

/hold

Which e2e tests are missing?

jmencak commented 7 months ago

/approved /label backport-risk-assessed Any reason some of the e2e tests are excluded from the manual backport? /hold

Which e2e tests are missing?

I've noticed the patch difference between 4.16/4.15 in these "testdata" files and wonder if this is intentional:

test/e2e/performanceprofile/testdata/render-expected-output/bootstrap/extra-ctrcfg/openshift-bootstrap-master_machineconfig.yaml
test/e2e/performanceprofile/testdata/render-expected-output/bootstrap/extra-ctrcfg/openshift-bootstrap-worker_machineconfig.yaml
test/e2e/performanceprofile/testdata/render-expected-output/default/cpuFrequency/manual_machineconfig.yaml
rbaturov commented 7 months ago

/approved /label backport-risk-assessed Any reason some of the e2e tests are excluded from the manual backport? /hold

Which e2e tests are missing?

I've noticed the patch difference between 4.16/4.15 in these "testdata" files and wonder if this is intentional:

test/e2e/performanceprofile/testdata/render-expected-output/bootstrap/extra-ctrcfg/openshift-bootstrap-master_machineconfig.yaml
test/e2e/performanceprofile/testdata/render-expected-output/bootstrap/extra-ctrcfg/openshift-bootstrap-worker_machineconfig.yaml
test/e2e/performanceprofile/testdata/render-expected-output/default/cpuFrequency/manual_machineconfig.yaml

Yes this was intentional. These testdata files are not part of the 4.15 release.

jmencak commented 7 months ago

/approve /lgtm

openshift-ci[bot] commented 7 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmencak, rbaturov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/cluster-node-tuning-operator/blob/release-4.15/OWNERS)~~ [jmencak] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
rbaturov commented 7 months ago

/approve /lgtm @jmencak Can we remove the hold label?

jmencak commented 7 months ago

/unhold

mrniranjan commented 7 months ago

/label cherry-pick-approved

openshift-ci-robot commented 7 months ago

/retest-required

Remaining retests: 0 against base HEAD 56014a8a39d47d5d6af431ff580c8ab90ea4967d and 2 for PR HEAD a2a2b522ddbe2a26778004039968dfafd6700b85 in total

openshift-ci[bot] commented 7 months ago

@rbaturov: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
openshift-ci-robot commented 7 months ago

@rbaturov: Jira Issue OCPBUGS-31692: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-31692 has been moved to the MODIFIED state.

In response to [this](https://github.com/openshift/cluster-node-tuning-operator/pull/1016): >This is a manual backport for #992 > >* Systemd processes not being moved to cpuset/systemd.slice fix > >The script cpuset-configure.sh is responsible to move the systemd processes to the cpuset/systemd.slice cgroup and is executed in a form of a service (cpuset-configure.service). > >In the current implementation, the script is executed too early - some system processes are yet to be created. >This in turn leads to them not being moved to the custom system slice. > >Moreover, in the current implementation, the script is executed before the network-online.target. The intention was to execute the script before kubelet and crio services are initialized (by the fact network-online.target is a common parent) in order to make sure that no workload pods are starting before we are making this transition. > >The fix I'm proposing consist of the following changes: >1. Adding an After statements - The script will start once crio service is initialized, due to the fact it's initialized in the very end of the boot process, just a bit before kubelet. >Thereby we can ensure late starting processes do not fall between the cracks. >2. Narrowing down the Before statement to a more accurate one, reflecting its original intention. (Running the script before kubelet only would be enough guarantee no workload pods are started at that time). > > > >* Added a test to verify system processes are in the correct cgroup > >When we are using cgroups v1 we are counting on the cpuset-configure.service to move all the system services to the custom system.slice. This test ensures the service indeed moved them. > >It is also a good practice to check for similar errors on cgroup v2 systems. > > > >--------- Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-node-tuning-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-bot commented 7 months ago

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-node-tuning-operator-container-v4.15.0-202404080709.p0.ge0ab1c1.assembly.stream.el9 for distgit cluster-node-tuning-operator. All builds following this will include this PR.

yanirq commented 6 months ago

/jira refresh

openshift-ci-robot commented 6 months ago

@yanirq: Jira Issue OCPBUGS-31692: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-31692 has been moved to the MODIFIED state.

In response to [this](https://github.com/openshift/cluster-node-tuning-operator/pull/1016#issuecomment-2085157431): >/jira refresh Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-node-tuning-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.