search

oras-project / oras

OCI registry client - managing content like artifacts, images, packages
https://oras.land
Apache License 2.0
1.33k stars 160 forks source link

Error trying to push a manifest to azure registry with retention policy #1382

Open side-chains opened 1 month ago

side-chains commented 1 month ago

What happened in your environment?

When trying to push a manifest with a tag to an azure registry with a retention policy, oras reports a 500: Internal Server Error. If I disable the retention policy on azure, I can push the tag successfully.

Interestingly, notation is able to push its blobs and manifests to the same registry, even when the retention policy is enabled. This suggests that there is something that can be done on the client side to make it work.

Notes:

The detailed output, with -d -v flags (I have added separator lines for visual comfort):

——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:28+02:00 level=debug msg=Request #0
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull\""
   "X-Content-Type-Options": "nosniff"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "46c27317-b52e-4c52-a266-455d332d6646"
   "Content-Length": "217"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #1
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #1
< Response Status: "200 OK"
< Response headers:
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "a68cf443-1d5f-4e66-8f5b-ff28eb9188f2"
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.316667"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #2
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #2
< Response Status: "404 Not Found"
< Response headers:
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Ms-Request-Id": "704e3e35-17f4-4667-9f23-78efa2de3a00"
   "Server": "AzureContainerRegistry"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "c09c0f21-233b-49ce-b8d1-885f73c533bd"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Length": "276"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Client-Request-Id": ""
Uploading 77f69ad47d16 application/vnd.oci.image.index.v1+json
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #3
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #3
< Response Status: "401 Unauthorized"
< Response headers:
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "21cc6879-d24f-4b6e-a7e3-e7d2bc932780"
   "Content-Length": "288"
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull,push\""
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #4
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #4
< Response Status: "200 OK"
< Response headers:
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.3"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "19fe5024-c9be-47c0-ad6f-cb7d28982426"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #5
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "User-Agent": "oras/1.1.0"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Authorization": "*****"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:51+02:00 level=debug msg=Response #5
< Response Status: "500 Internal Server Error"
< Response headers:
   "Content-Length": "0"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Ms-Client-Request-Id": ""
   "X-Ms-Request-Id": "7f778f02-d3c0-4206-aec8-69ad501a7d4a"
   "Server": "AzureContainerRegistry"
   "Connection": "keep-alive"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "07bea16f-8787-4ed8-8f16-d06aefe63ca7"
   "Date": "Mon, 13 May 2024 21:07:51 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
Error: failed to tag my-tag: PUT "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag": response status code 500: Internal Server Error

What did you expect to happen?

A successful response when pushing the tag.

How can we reproduce it?

Setup an azure registry and enable a retention policy. Then run

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo:my-tag -

What is the version of your ORAS CLI?

Version:        1.1.0
Go version:     go1.21.0
Git commit:     7079c468a06fb5815c99395eb4aaf46dd459d3fa
Git tree state: clean

What is your OS environment?

ubuntu 22.04

Are you willing to submit PRs to fix it?

qweeah commented 1 month ago

@side-chains How is the retention policy setup? What is the retention period?

qweeah commented 1 month ago

I have added separator lines for visual comfort

@FeynmanZhou A strong plus on adding such visual comfort to the HTTP trace logs.

side-chains commented 1 month ago

@side-chains How is the retention policy setup? What is the retention period?

Click on the registry resource. On the left menu, under policies, there is an item named retention (preview). There you can only enable or disable the retention policy and, if it is enabled, you can choose the retention period in days.

I have tried with registries with retention periods of 5 and 7 days. The problem is the same in both cases.

qweeah commented 1 month ago

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

side-chains commented 1 month ago

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

@qweeah Does oras use the same base code as notation? Or, does it not interact with the registry in a similar way? If the bug is on azure's side, how does it explain that notation can push and tag without getting the error?

qweeah commented 1 month ago

As far as I know, notation should generate artifact without tags.

side-chains commented 1 month ago

As far as I know, notation should generate artifact without tags.

@qweeah Well, it creates a signature manifest, with the subject being the image that is being signed, and one or more layers with signatures. This one is, indeed, untagged.

However, it also creates an index manifest with a list of manifests pointing to all signature manifests corresponding to that image. This one is tagged as sha256- followed by the sha256 of the image that is being signed.

qweeah commented 1 month ago

Okay, in that case(referrers tag schema) ORAS and notation should use the same SDK, oras-go. Will check the implementation detail later. Back to your issue, I am pretty sure that errors are caused by ACR but cannot share server-side logs here for privacy consideration.

qweeah commented 1 month ago

@side-chains I have confirmed that both Notation and ORAS push manifests in the same way. Your error is caused by a known issue in ACR retention policy feature(I cannot share server-side logs here but feel free to join our slack channel https://cloud-native.slack.com/archives/CJ1KHJM5Z to discuss). I would suggest you raise a ticket and get help from ACR supports directly.

sajayantony commented 3 weeks ago

@FeynmanZhou one more usecase IMHO remove -d and linking - #911