search

openshift / cloud-credential-operator

Manage cloud provider credentials as Kubernetes CRDs
Apache License 2.0
62 stars 143 forks source link

OCPBUGS-38951: Follow-up bug fixes for CCO-572 #745

Closed dlom closed 2 weeks ago

dlom commented 1 month ago

/assign @jstuever

openshift-ci-robot commented 1 month ago

@dlom: This pull request references CCO-572 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745): >/assign @jstuever Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 16.66667% with 5 lines in your changes missing coverage. Please review.

Project coverage is 47.27%. Comparing base (2c25ae6) to head (1637536). Report is 4 commits behind head on master.

Files Patch % Lines
pkg/gcp/actuator/actuator.go 16.66% 3 Missing and 2 partials :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/openshift/cloud-credential-operator/pull/745/graphs/tree.svg?width=650&height=150&src=pr&token=MSJhsyXrnA&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openshift)](https://app.codecov.io/gh/openshift/cloud-credential-operator/pull/745?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openshift) ```diff @@ Coverage Diff @@ ## master #745 +/- ## ========================================== - Coverage 47.28% 47.27% -0.02% ========================================== Files 96 96 Lines 11712 11717 +5 ========================================== + Hits 5538 5539 +1 - Misses 5563 5565 +2 - Partials 611 613 +2 ``` | [Files](https://app.codecov.io/gh/openshift/cloud-credential-operator/pull/745?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openshift) | Coverage Δ | | |---|---|---| | [pkg/gcp/actuator/actuator.go](https://app.codecov.io/gh/openshift/cloud-credential-operator/pull/745?src=pr&el=tree&filepath=pkg%2Fgcp%2Factuator%2Factuator.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openshift#diff-cGtnL2djcC9hY3R1YXRvci9hY3R1YXRvci5nbw==) | `50.36% <16.66%> (-0.29%)` | :arrow_down: |
dlom commented 1 month ago

/hold

dlom commented 1 month ago

/retest

2uasimojo commented 1 month ago

/lgtm

/retest

jianping-shu commented 1 month ago

@dlom I had a test on this PR. After creating the root credentials secret (work around for OCPBUGS-37952), it worked and the test case OCP-75429 was updated. one thing to mention is that CredentialsProvisionFailure set to True for the CredentialsRequest with audience etc. From the logs, it is the side effect for creating the credentials secret, not a real issue.

jianpingshu@jshu-mac ~ % oc -n openshift-cloud-credential-operator get -o json credentialsrequests | jq -r '.items[] | select(tostring | contains("InfrastructureMismatch") | not) | .metadata.name as $n | .status.conditions // [{type: "NoConditions"}] | .[] | .type + "=" + .status + " " + $n + " " + .reason + ": " + .message' | sort CredentialsProvisionFailure=False cloud-credential-operator-gcp-ro-creds CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-cloud-network-config-controller-gcp CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-gcp-ccm CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-gcp-pd-csi-driver-operator CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-image-registry-gcs CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-ingress-gcp CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=False openshift-machine-api-gcp CredentialsProvisionSuccess: successfully granted credentials request CredentialsProvisionFailure=True test-creds CredentialsProvisionFailure: failed to grant creds: error determining whether a credentials update is needed

logs: time="2024-08-12T00:59:53Z" level=debug msg="target secret exists" actuator=gcp cr=openshift-cloud-credential-operator/test-creds time="2024-08-12T00:59:53Z" level=error msg="cloud cred secret not yet annotated" actuator=gcp cr=openshift-cloud-credential-operator/test-creds secret=kube-system/gcp-credentials time="2024-08-12T00:59:53Z" level=debug msg="error retrieving cloud credentials secret" error="cannot proceed without cloud cred secret annotation" time="2024-08-12T00:59:53Z" level=error msg="error determining whether a credentials update is needed" actuator=gcp cr=openshift-cloud-credential-operator/test-creds error="cannot proceed without cloud cred secret annotation"

dlom commented 1 month ago

@jianping-shu have you tried creating read-only credentials as a workaround as well?

jianping-shu commented 1 month ago

@dlom I think the gcp read-only credential already exists after gcp wif cluster installs. But from cco logs, it still reported below. I'll double check this.

time="2024-08-11T03:46:11Z" level=warning msg="read-only creds not found, using root creds client" actuator=gcp cr=openshift-cloud-credential-operator/openshift-cloud-network-config-controller-gcp secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds

jianping-shu commented 4 weeks ago

@dlom Checked on the cluster. There's read-only creds but still log "read-only creds not found". Here is full log for one syncing. jianpingshu@jshu-mac ~ % oc get secret cloud-credential-operator-gcp-ro-creds -n openshift-cloud-credential-operator NAME TYPE DATA AGE cloud-credential-operator-gcp-ro-creds Opaque 1 6h35m

time="2024-08-15T00:35:55Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-image-registry-gcs time="2024-08-15T00:35:55Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=openshift-image-registry/installer-cloud-credentials time="2024-08-15T00:35:55Z" level=debug msg="timed token access cluster detected: true, so not trying to provision with root secret" controller=credreq cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=openshift-image-registry/installer-cloud-credentials time="2024-08-15T00:35:55Z" level=debug msg="target secret does not exist" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs time="2024-08-15T00:35:55Z" level=debug msg="running sync" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs time="2024-08-15T00:35:55Z" level=debug msg="Loading infrastructure name: jshu815-bbz9t" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs time="2024-08-15T00:35:55Z" level=debug msg="loading GCP read-only credentials from secret" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2024-08-15T00:35:55Z" level=warning msg="read-only creds not found, using root creds client" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds time="2024-08-15T00:35:55Z" level=debug msg="loading GCP credentials from secret" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=kube-system/gcp-credentials time="2024-08-15T00:35:55Z" level=error msg="error creating GCP client" error="Secret \"gcp-credentials\" not found" time="2024-08-15T00:35:55Z" level=error msg="error determining whether a credentials update is needed" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs error="unable to check whether credentialsRequest needs update"

dlom commented 4 weeks ago

@jianping-shu I have pushed another change to this branch that should fix the error. Please give it a try

jianping-shu commented 4 weeks ago

/retest

dlom commented 4 weeks ago

/retest

dlom commented 3 weeks ago

/retest

dlom commented 3 weeks ago

/retest

jianping-shu commented 3 weeks ago

@dlom I tested with cluster-bot build based on the new commit and it works fine now. Test case OCP-75429 - Test the STS OLM functionality of GCP has been updated. I also did simple regression test on mint/passthrough mode and no issue. But how about OCPBUGS-37952? Are you going to fix the bug by this PR for 4.18/4.17 and do manual back-porting for pre-4.17 releases?

dlom commented 3 weeks ago

/retest

dlom commented 3 weeks ago

@jianping-shu I plan to address OCPBUGS-37952 with a separate PR and backport it as necessary. I believe this PR (once tests pass) should merge ASAP.

dlom commented 3 weeks ago

/unhold

dlom commented 3 weeks ago

/assign @jstuever

dlom commented 3 weeks ago

/retest

dlom commented 3 weeks ago

/retest

dlom commented 3 weeks ago

/override ci/prow/okd-scos-images

openshift-ci[bot] commented 3 weeks ago

@dlom: Overrode contexts on behalf of dlom: ci/prow/okd-scos-images

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2302862736): >/override ci/prow/okd-scos-images Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
openshift-ci[bot] commented 3 weeks ago

@dlom: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
jstuever commented 2 weeks ago

/lgtm

openshift-ci[bot] commented 2 weeks ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo, dlom, jstuever

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/cloud-credential-operator/blob/master/OWNERS)~~ [2uasimojo,dlom,jstuever] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
dlom commented 2 weeks ago

/hold

dlom commented 2 weeks ago

/retitle OCPBUGS-38951: Follow-up bug fixes for CCO-572

openshift-ci-robot commented 2 weeks ago

@dlom: This pull request references Jira Issue OCPBUGS-38951, which is invalid:

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745): >/assign @jstuever Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
dlom commented 2 weeks ago

/jira refresh

openshift-ci-robot commented 2 weeks ago

@dlom: This pull request references Jira Issue OCPBUGS-38951, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.18.0) matches configured target version for branch (4.18.0) * bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact: /cc @jianping-shu

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310855342): >/jira refresh Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
dlom commented 2 weeks ago

/label acknowledge-critical-fixes-only

dlom commented 2 weeks ago

/jira backport release-4.17

openshift-ci-robot commented 2 weeks ago

@dlom: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges: /cherrypick release-4.17

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310857918): >/jira backport release-4.17 Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-cherrypick-robot commented 2 weeks ago

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.17 in a new PR and assign it to you.

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310858197): >@dlom: The following backport issues have been created: >- [OCPBUGS-38952](https://issues.redhat.com//browse/OCPBUGS-38952) for branch release-4.17 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.17 > >
> >In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310857918): > >>/jira backport release-4.17 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

dlom commented 2 weeks ago

/unhold

openshift-ci-robot commented 2 weeks ago

@dlom: Jira Issue OCPBUGS-38951: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-38951 has been moved to the MODIFIED state.

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745): >/assign @jstuever Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-cherrypick-robot commented 2 weeks ago

@openshift-ci-robot: new pull request created: #751

In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310858197): >@dlom: The following backport issues have been created: >- [OCPBUGS-38952](https://issues.redhat.com//browse/OCPBUGS-38952) for branch release-4.17 > >Queuing cherrypicks to the requested branches to be created after this PR merges: >/cherrypick release-4.17 > >
> >In response to [this](https://github.com/openshift/cloud-credential-operator/pull/745#issuecomment-2310857918): > >>/jira backport release-4.17 > > >Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcloud-credential-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository. >

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot commented 2 weeks ago

[ART PR BUILD NOTIFIER]

Distgit: ose-cloud-credential-operator This PR has been included in build ose-cloud-credential-operator-container-v4.18.0-202408270610.p0.gea71719.assembly.stream.el9. All builds following this will include this PR.