openshift-ci-robot
commented
4 months ago @wking: This pull request references Jira Issue OCPBUGS-37770, which is valid. The bug has been moved to the POST state.
3 validation(s) were run on this bug
* bug is open, matching expected state (open) * bug target version (4.17.0) matches configured target version for branch (4.17.0) * bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)The bug has been updated to refer to the pull request using the external bug tracker.
In response to [this](https://github.com/openshift/cluster-version-operator/pull/1076):
>Address [OCPBUGS-37770][1], where a ClusterImagePolicy manifest from the `cluster-update-keys` repository (openshift/cluster-update-keys#58) was not part of the bootstrap-rendered MachineConfigs, but was part of the production MachineConfigs. That kind of skew causes trouble for the machine-config operator, as in [this run][2]:
>
>```console
>$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-api-provider-aws/519/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-ovn-techpreview/1818614625273384960/artifacts/e2e-aws-ovn-techpreview/gather-extra/artifacts/machineconfigpools.json | jq -r '.items[0].status.conditions[] | select(.type == "NodeDegraded").message'
>Node ip-10-0-107-110.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found", Node ip-10-0-20-34.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found", Node ip-10-0-47-251.us-east-2.compute.internal is reporting: "missing MachineConfig rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\nmachineconfig.machineconfiguration.openshift.io \"rendered-master-f78e8c7637cfbc6ca42f0f102eaef4e7\" not found"
>```
>
>To address that, this commit renders any ClusterImagePolicy and ImagePolicy manifests from the release image into the output manifests directory. From there:
>
>1. The installer's bootkube service [puts the manifest into the central manifests directory][3]:
>
> ```sh
> cp cvo-bootstrap/manifests/* manifests/
> ```
>
>2. The installer's bootkube service [puts the manifest into into the machine-config controller's manifest directory][4]:
>
> ```sh
> cp manifests/* /etc/mcc/bootstrap/
> ```
>
>3. The MCO renders a static MCO bootstrap pod YAML file [with `--manifest-dir=/etc/mcc/bootstrap/`][5] to configure a `manifestDir` variable.
>
>4. The installer's bootkube service [passes the rendered static MCO bootstrap pod to the kubelet][6]:
>
> ```sh
> cp mco-bootstrap/bootstrap/machineconfigoperator-bootstrap-pod.yaml /etc/kubernetes/manifests/
> ```
>
>5. The bootstrap MCO static pod [pulls in the ClusterImagePolicy manifest while walking `manifestDir`][7].
>
>6. The bootstrap MCO static pod [includes feature-gate status and any collected ClusterImagePolicy manifest when rendering bootstrap MachineConfigs][8].
>
>My implementation uses group/kind tuples to figure out what needs rendering, because that is robust against things like manifest renames or other release-image-referenced repositories providing their own manifests (using manifest names in `skipFiles` is ok, because all the manifests we skip that way are from our own repository, where we control naming).
>
>We could extend this to other manifest types that the MCO consumes when rendering MachineConfigs (MachineConfig itself, KubeletConfig, etc.), but I'm leaving those out for now in the optimistic hope that nobody ever needs the CVO to manage those resources.
>
>[1]: https://issues.redhat.com/browse/OCPBUGS-37770
>[2]: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-api-provider-aws/519/pull-ci-openshift-cluster-api-provider-aws-master-e2e-aws-ovn-techpreview/1818614625273384960
>[3]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L165
>[4]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L431
>[5]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/manifests/bootstrap-pod-v2.yaml#L15
>[6]: https://github.com/openshift/installer/blob/e9c454889b638a9af3e1203d0585c756f8cf0136/data/data/bootstrap/files/usr/local/bin/bootkube.sh.template#L433
>[7]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/pkg/controller/bootstrap/bootstrap.go#L100-L142
>[8]: https://github.com/openshift/machine-config-operator/blob/d72d2ed139de8f8d3c60c411b4741045edde2a34/pkg/controller/bootstrap/bootstrap.go#L183
Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-version-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
wking
hongkailiu
openshift-ci[bot]
dis016
openshift-bot
Address OCPBUGS-37770, where a ClusterImagePolicy manifest from the
cluster-update-keysrepository (openshift/cluster-update-keys#58) was not part of the bootstrap-rendered MachineConfigs, but was part of the production MachineConfigs. That kind of skew causes trouble for the machine-config operator, as in this run:To address that, this commit renders any ClusterImagePolicy and ImagePolicy manifests from the release image into the output manifests directory. From there:
The installer's bootkube service puts the manifest into the central manifests directory:
The installer's bootkube service puts the manifest into into the machine-config controller's manifest directory:
The MCO renders a static MCO bootstrap pod YAML file with
--manifest-dir=/etc/mcc/bootstrap/to configure amanifestDirvariable.The installer's bootkube service passes the rendered static MCO bootstrap pod to the kubelet:
The bootstrap MCO static pod pulls in the ClusterImagePolicy manifest while walking
manifestDir.The bootstrap MCO static pod includes feature-gate status and any collected ClusterImagePolicy manifest when rendering bootstrap MachineConfigs.
My implementation uses group/kind tuples to figure out what needs rendering, because that is robust against things like manifest renames or other release-image-referenced repositories providing their own manifests (using manifest names in
skipFilesis ok, because all the manifests we skip that way are from our own repository, where we control naming).We could extend this to other manifest types that the MCO consumes when rendering MachineConfigs (MachineConfig itself, KubeletConfig, etc.), but I'm leaving those out for now in the optimistic hope that nobody ever needs the CVO to manage those resources.