Unable to pull latest stacks from community staging deployment

michael-valdron commented 1 year ago

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area registry

What versions of software are you using?

Go project

Operating System and version: N/A

Go Pkg Version: 1.17

Bug Summary

Describe the bug:

While using odo version 3 against our community registry staging deployment, there is an error when trying to fetch the latest version of the stack. The error thrown appears to be a not found error type when the client attempts to fetch content descriptors or the manifest from the OCI registry.

This error happens at inconsistent attempts and seems to successfully fetch the stack at inconsistent number of tries. After trying on other deployments, this seems to be isolated to the community staging deployment.

To Reproduce:

Run the following commands with odo version 3:

odo preference add registry stage https://registry.stage.devfile.io

go stack

odo init --name aname --devfile go --devfile-version latest --devfile-registry stage

python stack

odo init --name aname --devfile python --devfile-version latest --devfile-registry stage

java-springboot stack

odo init --name aname --devfile java-springboot --devfile-version latest --devfile-registry stage

Expected behavior

The output of any of these odo init commands should be something like the following:

  __
 /  \__     Initializing a new component
 \__/  \    
 /  \__/    odo version: v3.8.0
 \__/

 ✓  Downloading devfile "go:latest" from registry "stage" [543ms]

Your new component 'aname' is ready in the current directory.
To start editing your component, use 'odo dev' and open this folder in your favorite IDE.
Changes will be directly reflected on the cluster.
To deploy your component to a cluster use "odo deploy".

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable

Manifest not found error output

  __
 /  \__     Initializing a new component
 \__/  \    
 /  \__/    odo version: v3.8.0
 \__/

 ✗  Downloading devfile "go:latest" from registry "stage" [296ms]
 ✗  unable to download devfile: failed to pull stack go:latest from registry.stage.devfile.io/devfile-catalog/go:2.0.0 with allowed media types [application/vnd.devfileio.devfile.layer.v1 image/png image/svg+xml application/vnd.devfileio.vsx.layer.v1.tar application/x-tar]: failed to copy: httpReadSeeker: failed open: content at https://registry.stage.devfile.io/v2/devfile-catalog/go/manifests/sha256:c15d231dc4ce9a82f787f91390a066f1edfcc611c016fd8748dc65a29287ba3b not found: not found
the command failed, the devfile has been removed from current directory

Content descriptor error output

  __
 /  \__     Initializing a new component
 \__/  \    
 /  \__/    odo version: v3.8.0
 \__/

 ✗  Downloading devfile "go:latest" from registry "stage" [416ms]
 ✗  unable to download devfile: failed to pull stack go:latest from registry.stage.devfile.io/devfile-catalog/go:2.0.0 with allowed media types [application/vnd.devfileio.devfile.layer.v1 image/png image/svg+xml application/vnd.devfileio.vsx.layer.v1.tar application/x-tar]: failed to copy: httpReadSeeker: failed open: could not fetch content descriptor sha256:013466fc4eed3c375821002e5e7e80939aa848b7fb86d5b6b4b131422b29b146 (application/x-tar) from remote: not found
the command failed, the devfile has been removed from current directory

Server Log

[GIN] 2023/03/24 - 17:41:47 | 200 | 1.349969ms | x.x.x.x | GET "/v2index"
[GIN] 2023/03/24 - 17:41:47 | 200 | 2.465988ms | x.x.x.x | HEAD "/v2/devfile-catalog/go/manifests/2.0.0"

Additional context

Any workaround?

No official workarounds have been found, however since the errors do not happen every time it can be retried for a successful fetch.

Suggestion on how to fix the bug

An update to the image tag for the OCI registry container: https://github.com/devfile/api/issues/1078#issuecomment-1483285440

Target Date: 03-27-2023

michael-valdron commented 1 year ago

Devfile registry server logs do not show any error has occurred, however does miss request logs when client errors which shows when client does a successful fetch of the stack.

michael-valdron commented 1 year ago

After further investigation I found that the image tag for the OCI registry container used on our staging deployment was a year outdated. This image would lack the sufficient security permissions needed for the required setup security context definitions: https://github.com/devfile/registry/blob/47eee922b60b5a90e0fbcbf9966300c990127de4/.ci/deploy/devfile-registry.yaml#L142

Since updating this image tag we have noticed no more occurrences of these errors.

michael-valdron commented 1 year ago

Updating the image tag has been confirmed to resolve this issue, closing now.

devfile / api