search

openshift / sriov-network-operator

SR-IOV Network Operator
Apache License 2.0
119 stars 106 forks source link

OCPBUGS-43027: [release-4.17] config-daemon: Restart all instances of device-plugin #1015

Closed zeeke closed 1 month ago

zeeke commented 1 month ago

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing. If the config-daemon executes restartDevicePluginPod() at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

4.17 backport of

zeeke commented 1 month ago

/jira cherrypick OCPBUGS-41356

openshift-ci-robot commented 1 month ago

@zeeke: Jira Issue OCPBUGS-41356 has been cloned as Jira Issue OCPBUGS-43027. Will retitle bug to link to clone. /retitle OCPBUGS-43027: config-daemon: Restart all instances of device-plugin

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404789008): >/jira cherrypick OCPBUGS-41356 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci-robot commented 1 month ago

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015): >When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing. >If the config-daemon executes `restartDevicePluginPod()` at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs. > >Make the config-daemon restart all the device-plugin instances it founds for its own node. > >4.17 backport of >- https://github.com/k8snetworkplumbingwg/sriov-network-operator/pull/783 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
zeeke commented 1 month ago

/jira backport release-4.16,release-4.15,release-4.14

openshift-ci-robot commented 1 month ago

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges: /cherrypick release-4.16 /cherrypick release-4.15 /cherrypick release-4.14

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404790527): >/jira backport release-4.16,release-4.15,release-4.14 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-cherrypick-robot commented 1 month ago

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.16 in a new PR and assign it to you.

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404791343): >@zeeke: The following backport issues have been created: >- [OCPBUGS-43029](https://issues.redhat.com//browse/OCPBUGS-43029) for branch release-4.16 >- [OCPBUGS-43030](https://issues.redhat.com//browse/OCPBUGS-43030) for branch release-4.15 >- [OCPBUGS-43031](https://issues.redhat.com//browse/OCPBUGS-43031) for branch release-4.14 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.16 >/cherrypick release-4.15 >/cherrypick release-4.14 > >
> >In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404790527): > >>/jira backport release-4.16,release-4.15,release-4.14 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zeeke commented 1 month ago

/test e2e-telco5g-sriov

zeeke commented 1 month ago

/retest

zeeke commented 1 month ago

/jira refresh

@evgenLevin @bn222 @wizhaoredhat please take a look at this backport

openshift-ci-robot commented 1 month ago

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2419965599): >/jira refresh > >@evgenLevin @bn222 @wizhaoredhat please take a look at this backport Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
wizhaoredhat commented 1 month ago

/lgtm

openshift-ci[bot] commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wizhaoredhat, zeeke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/sriov-network-operator/blob/release-4.17/OWNERS)~~ [wizhaoredhat,zeeke] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
zeeke commented 1 month ago

/jira refresh /label backport-risk-assessed @evgenLevin @ajaggapa when you have time, take a look at this backport. I need cherry-pick-approved label

openshift-ci-robot commented 1 month ago

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.17.z) matches configured target version for branch (4.17.z) * bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST) * release note text is set and does not match the template * dependent bug [Jira Issue OCPBUGS-41356](https://issues.redhat.com//browse/OCPBUGS-41356) is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA)) * dependent [Jira Issue OCPBUGS-41356](https://issues.redhat.com//browse/OCPBUGS-41356) targets the "4.18.0" version, which is one of the valid target versions: 4.18.0 * bug has dependents

Requesting review from QA contact: /cc @evgenLevin

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2426006002): >/jira refresh >/label backport-risk-assessed >@evgenLevin @ajaggapa when you have time, take a look at this backport. I need `cherry-pick-approved` label > > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
evgenLevin commented 1 month ago

/label cherry-pick-approved

openshift-ci[bot] commented 1 month ago

@zeeke: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
openshift-ci-robot commented 1 month ago

@zeeke: Jira Issue OCPBUGS-43027: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-43027 has been moved to the MODIFIED state.

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015): >When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing. >If the config-daemon executes `restartDevicePluginPod()` at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs. > >Make the config-daemon restart all the device-plugin instances it founds for its own node. > >4.17 backport of >- https://github.com/k8snetworkplumbingwg/sriov-network-operator/pull/783 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-cherrypick-robot commented 1 month ago

@openshift-ci-robot: new pull request created: #1023

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404791343): >@zeeke: The following backport issues have been created: >- [OCPBUGS-43029](https://issues.redhat.com//browse/OCPBUGS-43029) for branch release-4.16 >- [OCPBUGS-43030](https://issues.redhat.com//browse/OCPBUGS-43030) for branch release-4.15 >- [OCPBUGS-43031](https://issues.redhat.com//browse/OCPBUGS-43031) for branch release-4.14 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.16 >/cherrypick release-4.15 >/cherrypick release-4.14 > >
> >In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404790527): > >>/jira backport release-4.16,release-4.15,release-4.14 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-cherrypick-robot commented 1 month ago

@openshift-ci-robot: new pull request created: #1024

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404791343): >@zeeke: The following backport issues have been created: >- [OCPBUGS-43029](https://issues.redhat.com//browse/OCPBUGS-43029) for branch release-4.16 >- [OCPBUGS-43030](https://issues.redhat.com//browse/OCPBUGS-43030) for branch release-4.15 >- [OCPBUGS-43031](https://issues.redhat.com//browse/OCPBUGS-43031) for branch release-4.14 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.16 >/cherrypick release-4.15 >/cherrypick release-4.14 > >
> >In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404790527): > >>/jira backport release-4.16,release-4.15,release-4.14 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-cherrypick-robot commented 1 month ago

@openshift-ci-robot: new pull request created: #1025

In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404791343): >@zeeke: The following backport issues have been created: >- [OCPBUGS-43029](https://issues.redhat.com//browse/OCPBUGS-43029) for branch release-4.16 >- [OCPBUGS-43030](https://issues.redhat.com//browse/OCPBUGS-43030) for branch release-4.15 >- [OCPBUGS-43031](https://issues.redhat.com//browse/OCPBUGS-43031) for branch release-4.14 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.16 >/cherrypick release-4.15 >/cherrypick release-4.14 > >
> >In response to [this](https://github.com/openshift/sriov-network-operator/pull/1015#issuecomment-2404790527): > >>/jira backport release-4.16,release-4.15,release-4.14 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fsriov-network-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot commented 1 month ago

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-config-daemon This PR has been included in build sriov-network-config-daemon-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9. All builds following this will include this PR.

openshift-bot commented 1 month ago

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-operator This PR has been included in build sriov-network-operator-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9. All builds following this will include this PR.

openshift-bot commented 1 month ago

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-webhook This PR has been included in build sriov-network-webhook-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9. All builds following this will include this PR.