Failing to release due to v4.16 index image

[tiraboschi]

Trying to release something with:

  com.redhat.openshift.versions: "v4.15"

(see https://github.com/redhat-openshift-ecosystem/community-operators-prod/pull/4138 ) for instance, is currently failing to to the lack of permissions on the v4.16 index image.

see:

[get-supported-versions : supported-version-check] {
[get-supported-versions : supported-version-check]   "versions_annotation": "v4.15",
[get-supported-versions : supported-version-check]   "indices": [
[get-supported-versions : supported-version-check]     {
[get-supported-versions : supported-version-check]       "end_of_life": "2028-07-21T08:40:42.484000+00:00",
[get-supported-versions : supported-version-check]       "ocp_version": "4.16",
[get-supported-versions : supported-version-check]       "path": "registry.redhat.io/redhat/community-operator-index:v4.16"
[get-supported-versions : supported-version-check]     },
[get-supported-versions : supported-version-check]     {
[get-supported-versions : supported-version-check]       "ocp_version": "4.15",
[get-supported-versions : supported-version-check]       "path": "registry.redhat.io/redhat/community-operator-index:v4.15"
[get-supported-versions : supported-version-check]     }
[get-supported-versions : supported-version-check]   ],
[get-supported-versions : supported-version-check]   "max_version_index": {
[get-supported-versions : supported-version-check]     "end_of_life": "2028-07-21T08:40:42.484000+00:00",
[get-supported-versions : supported-version-check]     "ocp_version": "4.16",
[get-supported-versions : supported-version-check]     "path": "registry.redhat.io/redhat/community-operator-index:v4.16"
[get-supported-versions : supported-version-check]   },
....

and:

[add-bundle-to-index : add-bundle-to-index] + ENV=prod
[add-bundle-to-index : add-bundle-to-index] + INDEX_IMAGES=registry.redhat.io/redhat/community-operator-index:v4.16
[add-bundle-to-index : add-bundle-to-index] + [[ prod == \d\e\v ]]
[add-bundle-to-index : add-bundle-to-index] + [[ prod == \q\a ]]
[add-bundle-to-index : add-bundle-to-index] + [[ prod != \p\r\o\d ]]
[add-bundle-to-index : add-bundle-to-index] + EXTRA_ARGS=
[add-bundle-to-index : add-bundle-to-index] + [[ true == \t\r\u\e ]]
[add-bundle-to-index : add-bundle-to-index] + EXTRA_ARGS+=' --authfile /workspace/credentials/.dockerconfigjson'
[add-bundle-to-index : add-bundle-to-index] + [[ -8fa00b39fb75a9629865fc67b1ee462601c52927 != '' ]]
[add-bundle-to-index : add-bundle-to-index] + EXTRA_ARGS+=' --index-image-destination-tag-suffix=-8fa00b39fb75a9629865fc67b1ee462601c52927'
[add-bundle-to-index : add-bundle-to-index] + [[ semver != '' ]]
[add-bundle-to-index : add-bundle-to-index] + EXTRA_ARGS+=' --mode semver'
[add-bundle-to-index : add-bundle-to-index] + index --iib-url https://iib.engineering.redhat.com --indices registry.redhat.io/redhat/community-operator-index:v4.16 --bundle-pullspec quay.io/community-operator-pipeline-prod/community-kubevirt-hyperconverged:1.11.0-8fa00b39fb75a9629865fc67b1ee462601c52927 --index-image-destination quay.io/community-operator-pipeline-prod/catalog --image-output index-image-paths.txt --authfile /workspace/credentials/.dockerconfigjson --index-image-destination-tag-suffix=-8fa00b39fb75a9629865fc67b1ee462601c52927 --mode semver
[add-bundle-to-index : add-bundle-to-index] 2024-03-06 15:35:10,562 [operator-cert] ERROR IIB POST query failed with https://iib.engineering.redhat.com/api/v1/builds/add-rm-batch - 403 - {"error":"\"graph_update_mode\" can only be used on the following index image: ['registry.stage.redhat.io/redhat/community-operator-index:v4.10', 'registry.stage.redhat.io/redhat/community-operator-index:v4.11', 'registry.stage.redhat.io/redhat/community-operator-index:v4.12', 'registry.stage.redhat.io/redhat/community-operator-index:v4.13', 'registry.stage.redhat.io/redhat/community-operator-index:v4.14', 'registry.stage.redhat.io/redhat/community-operator-index:v4.15', 'registry.redhat.io/redhat/community-operator-index:v4.10', 'registry.redhat.io/redhat/community-operator-index:v4.11', 'registry.redhat.io/redhat/community-operator-index:v4.12', 'registry.redhat.io/redhat/community-operator-index:v4.13', 'registry.redhat.io/redhat/community-operator-index:v4.14', 'registry.redhat.io/redhat/community-operator-index:v4.15', 'registry-proxy-stage.engineering.redhat.com/rh-osbs-stage/iib-qe:empty-iib-pub-pending-v4.9', 'registry-proxy-stage.engineering.redhat.com/rh-osbs-stage/iib-qe:empty-iib-pub-pending-v4.14']"}
[add-bundle-to-index : add-bundle-to-index] Traceback (most recent call last):
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/operatorcert/iib.py", line 51, in add_builds
[add-bundle-to-index : add-bundle-to-index]     resp.raise_for_status()
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
[add-bundle-to-index : add-bundle-to-index]     raise HTTPError(http_error_msg, response=self)
[add-bundle-to-index : add-bundle-to-index] requests.exceptions.HTTPError: 403 Client Error: FORBIDDEN for url: https://iib.engineering.redhat.com/api/v1/builds/add-rm-batch
[add-bundle-to-index : add-bundle-to-index] Traceback (most recent call last):
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/bin/index", line 8, in <module>
[add-bundle-to-index : add-bundle-to-index]     sys.exit(main())
[add-bundle-to-index : add-bundle-to-index]              ^^^^^^
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/operatorcert/entrypoints/index.py", line 234, in main
[add-bundle-to-index : add-bundle-to-index]     iib_response = add_bundle_to_index(
[add-bundle-to-index : add-bundle-to-index]                    ^^^^^^^^^^^^^^^^^^^^
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/operatorcert/entrypoints/index.py", line 153, in add_bundle_to_index
[add-bundle-to-index : add-bundle-to-index]     resp = iib.add_builds(iib_url, payload)
[add-bundle-to-index : add-bundle-to-index]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/operatorcert/iib.py", line 51, in add_builds
[add-bundle-to-index : add-bundle-to-index]     resp.raise_for_status()
[add-bundle-to-index : add-bundle-to-index]   File "/home/user/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
[add-bundle-to-index : add-bundle-to-index]     raise HTTPError(http_error_msg, response=self)
[add-bundle-to-index : add-bundle-to-index] requests.exceptions.HTTPError: 403 Client Error: FORBIDDEN for url: https://iib.engineering.redhat.com/api/v1/builds/add-rm-batch

in https://gist.github.com/rh-operator-bundle-bot/487ec129fa4bef61916b5b04c465eb18

[tiraboschi]

Many operators are now impaceted: #4148 , #4147 , #4137 ...

[Allda]

Hey @tiraboschi thank you for reporting the issue. We are aware of this issue and we are working on a fix now. The fix should be ready in a couple of minutes. After that, I am going to restart the failed pipelines.

[Allda]

All failed PRs affected by the 4.16 issue have been fixed. You you still see some PRs feel free to reopen this issue.