search

oras-project / oras

OCI registry client - managing content like artifacts, images, packages
https://oras.land
Apache License 2.0
1.44k stars 174 forks source link

oras is slow on push #923

Open rachelwaldon opened 1 year ago

rachelwaldon commented 1 year ago

What happened in your environment?

@srkohn Trying to push a .sif file to the GitLab container registry takes a very long time measured around 1 hour in my GitLab job. The size of the .sif is 580 MB. This is too slow for me to use in a GitLab pipeline?

err

What did you expect to happen?

I would expect the upload rate to be faster?

How can we reproduce it?

pushing a medium sized .sif file 580 MB with the oras push command to the GitLab registry

$ oras push ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA} pingpong.sif

What is the version of your ORAS CLI?

Version: 1.0.0 Go version: go1.20.2 Git commit: b58e7b910ca556973d111e9bd734a71baef03db2 Git tree state: clean

What is your OS environment?

Ubuntu 22.04

Are you willing to submit PRs to fix it?

TerryHowe commented 1 year ago

Any chance you could add -v -d options to that to see where it is stuck?

rachelwaldon commented 1 year ago

Here is the output of running with -v and -d flags. This run took 18 minutes. Anyway to get that time down?

$ oras push -d -v ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA} pingpong.sif
Preparing pingpong.sif
time=2023-04-10T16:23:20-07:00 level=debug msg=Request #0
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/manifests/sha256:4a69576e405f6839bf49b7e113c035d0f9a11e40746112cf02a78dda77742274"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:21-07:00 level=debug msg=Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Content-Type": "application/json"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Www-Authenticate": "Bearer realm=\"https://gitlab.com/jwt/auth\",service=\"container_registry\",scope=\"repository:rwaldon/pingpong/pingpong:pull\""
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:23:21 GMT"
   "Content-Length": "168"
time=2023-04-10T16:23:21-07:00 level=debug msg=Request #1
> Request URL: "https://gitlab.com/jwt/auth?scope=repository%3Arwaldon%2Fpingpong%2Fpingpong%3Apull&service=container_registry"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:21-07:00 level=debug msg=Response #1
< Response Status: "200 OK"
< Response headers:
   "Content-Security-Policy": "base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/ https://gitlab.com/assets/ blob: data:; connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com snowplow.trx.gitlab.net; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-cloudresourcemanager.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://*.codesandbox.io https://customers.gitlab.com https://*.zuora.com/apps/PublicHostedPageLite.do; img-src * data: blob:; manifest-src 'self'; media-src 'self' data: http: https:; object-src 'none'; report-uri https://sentry.gitlab.net/api/105/security/?sentry_key=a42ea3adc19140d9a6424906e12fba86; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.recaptcha.net/ https://apis.google.com https://*.zuora.com/apps/PublicHostedPageLite.do 'nonce-eAfwrdNJuz7hDJilU4zFHA=='; style-src 'self' 'unsafe-inline'; worker-src https://gitlab.com blob: data:; form-action 'self' https: http: vscode:"
   "Etag": "W/\"ac463e58809cb48b114b113cb3048626\""
   "X-Content-Type-Options": "nosniff"
   "Cf-Cache-Status": "MISS"
   "Vary": "Accept-Encoding"
   "Set-Cookie": "_cfuvid=Z5U3IYv8FJ9fKD4ECShmGBZUhvr15_HRnKgVMI8itU8-1681169001764-0-[60](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L60)4800000; path=/; domain=.gitlab.com; HttpOnly; Secure; SameSite=None"
   "Cf-Ray": "7b5ebcb3f819cefd-SJC"
   "Date": "Mon, 10 Apr 2023 23:23:21 GMT"
   "Referrer-Policy": "strict-origin-when-cross-origin"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Request-Id": "01GXPQPB6G047QNTFZ2PGR6WSQ"
   "Gitlab-Lb": "fe-02-lb-gprd"
   "Nel": "{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}"
   "Strict-Transport-Security": "max-age=31536000"
   "X-Download-Options": "noopen"
   "X-Permitted-Cross-Domain-Policies": "none"
   "X-Runtime": "0.045463"
   "X-Xss-Protection": "1; mode=block"
   "Content-Type": "application/json; charset=utf-8"
   "Cache-Control": "max-age=0, private, must-revalidate"
   "Permissions-Policy": "interest-cohort=()"
   "X-Ua-Compatible": "IE=edge"
   "Gitlab-Sv": "web-gke-us-east1-d"
   "Report-To": "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=vbk5g%2FLUIA0J9DXxsjpntJeo5XtMMwzChnHGZbeJms8L6Kj20EOoANwJxwwwuvcZU1%2BvFuwRBE4uS44jlrfXl4YynZZ8ED4fFqwl0HZGHfpqAYkiNAV4d9zxv8U%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}"
   "Server": "cloudflare"
time=2023-04-10T16:23:21-07:00 level=debug msg=Request #2
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/manifests/sha256:4a69576e405f6839bf49b7e113c035d0f9a11e4074[61](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L61)12cf02a78dda77742274"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:21-07:00 level=debug msg=Response #2
< Response Status: "404 Not Found"
< Response headers:
   "Date": "Mon, 10 Apr 2023 23:23:21 GMT"
   "Content-Length": "201"
   "Content-Type": "application/json"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
time=2023-04-10T16:23:21-07:00 level=debug msg=Request #4
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
> Request method: "HEAD"
> Request headers:
   "User-Agent": "oras/1.0.0"
   "Authorization": "*****"
time=2023-04-10T16:23:21-07:00 level=debug msg=Request #3
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/sha256:76c429b63adb164a2733ee7c7ffd2ebbd6b26a490e291286fcf8fae381af8fe1"
> Request method: "HEAD"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:22-07:00 level=debug msg=Response #4
< Response Status: "307 Temporary Redirect"
< Response headers:
   "Content-Type": "text/html; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Location": "https://cdn.registry.gitlab-static.net/gitlab/docker/registry/v2/blobs/sha256/44/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/data?Expires=1681170202&KeyName=gprd-registry-cdn&Signature=qV4eb_tLTRkJqSF-xQJhRu_Tivo="
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:23:22 GMT"
time=2023-04-10T16:23:22-07:00 level=debug msg=Request #5
> Request URL: "https://cdn.registry.gitlab-static.net/gitlab/docker/registry/v2/blobs/sha256/44/44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a/data?Expires=1681170202&KeyName=gprd-registry-cdn&Signature=qV4eb_tLTRkJqSF-xQJhRu_Tivo="
> Request method: "HEAD"
> Request headers:
   "User-Agent": "oras/1.0.0"
   "Referer": "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
time=2023-04-10T16:23:22-07:00 level=debug msg=Response #3
< Response Status: "404 Not Found"
< Response headers:
   "Content-Type": "application/json"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:23:22 GMT"
   "Content-Length": "157"
Uploading 76c429b63adb pingpong.sif
time=2023-04-10T16:23:22-07:00 level=debug msg=Request #6
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/uploads/"
> Request method: "POST"
> Request headers:
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:22-07:00 level=debug msg=Response #6
< Response Status: "401 Unauthorized"
< Response headers:
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Www-Authenticate": "Bearer realm=\"https://gitlab.com/jwt/auth\",service=\"container_registry\",scope=\"repository:rwaldon/pingpong/pingpong:pull,push\""
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:23:22 GMT"
   "Content-Length": "252"
   "Content-Type": "application/json"
time=2023-04-10T16:23:22-07:00 level=debug msg=Request #7
> Request URL: "https://gitlab.com/jwt/auth?scope=repository%3Arwaldon%2Fpingpong%2Fpingpong%3Apull%2Cpush&service=container_registry"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:22-07:00 level=debug msg=Response #5
< Response Status: "200 OK"
< Response headers:
   "X-Goog-Storage-Class": "MULTI_REGIONAL"
   "Accept-Ranges": "bytes"
   "X-Goog-Hash": "crc32c=KXvQqg==, md5=mZFLkyvTelC5g8XnyQrpOw=="
   "X-Goog-Generation": "1646326437088720"
   "X-Goog-Stored-Content-Encoding": "identity"
   "X-Goog-Stored-Content-Length": "2"
   "Server": "UploadServer"
   "Date": "Mon, 10 Apr 2023 23:23:22 GMT"
   "Etag": "\"99914b932bd37a50b983c5e7c90ae93b\""
   "X-Goog-Metageneration": "1"
   "Content-Length": "2"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Last-Modified": "Thu, 03 Mar 2022 16:53:57 GMT"
   "Content-Type": "application/octet-stream"
   "Cache-Control": "private,max-age=0,public"
   "X-Guploader-Uploadid": "ADPycdtmT-vGv7PJTXaHzApilC2VelmBL94EdWtsWHlUUE1QfaDzx3DyrMAg3Z9Kg2zvJ35RlaVyooer3Y3vWTA4jvVbXA"
Exists    44136fa355b3 application/vnd.unknown.config.v1+json
time=2023-04-10T16:23:22-07:00 level=debug msg=Response #7
< Response Status: "200 OK"
< Response headers:
   "Cf-Ray": "7b5ebcb86e9ccefd-SJC"
   "Date": "Mon, 10 Apr 2023 23:23:22 GMT"
   "Cache-Control": "max-age=0, private, must-revalidate"
   "X-Download-Options": "noopen"
   "X-Xss-Protection": "1; mode=block"
   "Vary": "Accept-Encoding"
   "Set-Cookie": "_cfuvid=ezr.0CEb2JZ5mwHlZDMScmeOBcqJCCKObcp_dVwbaj0-1681169002494-0-604800000; path=/; domain=.gitlab.com; HttpOnly; Secure; SameSite=None"
   "Nel": "{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}"
   "Server": "cloudflare"
   "Content-Type": "application/json; charset=utf-8"
   "X-Permitted-Cross-Domain-Policies": "none"
   "X-Request-Id": "01GXPQPBWF1TJEV4W73FH2090Y"
   "X-Ua-Compatible": "IE=edge"
   "Cf-Cache-Status": "MISS"
   "Report-To": "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=eUMg07tT96ojh1EG%2FPfhxWkQ5Hwu5MI0U5jNWcl22XfV0%2BxAyI%2B0SB6D7%2BYynrmE4iZGdacDIUiq%2BVTnE0oT2P1xPLUcO91g%2FMi%2B4mL5CBRJ23UlYi2OfLjl04o%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}"
   "Strict-Transport-Security": "max-age=31536000"
   "Content-Security-Policy": "base-uri 'self'; child-src https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://content-cloudresourcemanager.googleapis.com https://www.googletagmanager.com/ns.html https://*.zuora.com/apps/PublicHostedPageLite.do https://gitlab.com/admin/ https://gitlab.com/assets/ https://gitlab.com/-/speedscope/index.html https://gitlab.com/-/sandbox/ https://gitlab.com/assets/ blob: data:; connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com snowplow.trx.gitlab.net; default-src 'self'; font-src 'self'; frame-ancestors 'self'; frame-src 'self' https://www.google.com/recaptcha/ https://www.recaptcha.net/ https://content.googleapis.com https://content-cloudresourcemanager.googleapis.com https://content-compute.googleapis.com https://content-cloudbilling.googleapis.com https://*.codesandbox.io https://customers.gitlab.com https://*.zuora.com/apps/PublicHostedPageLite.do; img-src * data: blob:; manifest-src 'self'; media-src 'self' data: http: https:; object-src 'none'; report-uri https://sentry.gitlab.net/api/105/security/?sentry_key=a42ea3adc19140d9a6424906e12fba86; script-src 'strict-dynamic' 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://www.recaptcha.net/ https://apis.google.com https://*.zuora.com/apps/PublicHostedPageLite.do 'nonce-ucbEWh25BABE3bAj/9ja9w=='; style-src 'self' 'unsafe-inline'; worker-src https://gitlab.com blob: data:; form-action 'self' https: http: http:"
   "Etag": "W/\"3ac8dd4b15c684d30b746bf90d9ff730\""
   "X-Frame-Options": "SAMEORIGIN"
   "X-Runtime": "0.068488"
   "Gitlab-Lb": "fe-08-lb-gprd"
   "Gitlab-Sv": "web-gke-us-east1-d"
   "Permissions-Policy": "interest-cohort=()"
   "Referrer-Policy": "strict-origin-when-cross-origin"
   "X-Content-Type-Options": "nosniff"
time=2023-04-10T16:23:22-07:00 level=debug msg=Request #8
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/uploads/"
> Request method: "POST"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:23:23-07:00 level=debug msg=Response #8
< Response Status: "202 Accepted"
< Response headers:
   "Date": "Mon, 10 Apr 2023 23:23:23 GMT"
   "Content-Length": "0"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Docker-Upload-Uuid": "73503ffa-c0fd-4c5c-9[62](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L62)b-8f1b8cd8011b"
   "Location": "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/uploads/73503ffa-c0fd-4c5c-962b-8f1b8cd8011b?_state=Ak4-7A6sKZo7ln88Gh4o6mheYMjTjeJL6Rh-93JT23B7Ik5hbWUiOiJyd2FsZG9uL3Bpbmdwb25nL3Bpbmdwb25nIiwiVVVJRCI6IjczNTAzZmZhLWMwZmQtNGM1Yy05NjJiLThmMWI4Y2Q4MDExYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0wNC0xMFQyMzoyMzoyMi42ODM5OTU4NzhaIn0%3D"
   "Range": "0-0"
   "X-Content-Type-Options": "nosniff"
time=2023-04-10T16:23:23-07:00 level=debug msg=Request #9
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/uploads/73503ffa-c0fd-4c5c-962b-8f1b8cd8011b?_state=Ak4-7A6sKZo7ln88Gh4o6mheYMjTjeJL6Rh-93JT23B7Ik5hbWUiOiJyd2FsZG9uL3Bpbmdwb25nL3Bpbmdwb25nIiwiVVVJRCI6IjczNTAzZmZhLWMwZmQtNGM1Yy05NjJiLThmMWI4Y2Q4MDExYiIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0wNC0xMFQyMzoyMzoyMi42ODM5OTU4NzhaIn0%3D&digest=sha256%3A76c429b63adb164a2733ee7c7ffd2ebbd6b26a490e291286fcf8fae381af8fe1"
> Request method: "PUT"
> Request headers:
   "Content-Type": "application/octet-stream"
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:37:19-07:00 level=debug msg=Response #9
< Response Status: "201 Created"
< Response headers:
   "Content-Length": "0"
   "Docker-Content-Digest": "sha256:76c429b63adb164a2733ee7c7ffd2ebbd6b26a490e291286fcf8fae381af8fe1"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Location": "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/blobs/sha256:76c429b63adb164a2733ee7c7ffd2ebbd6b26a490e291286fcf8fae381af8fe1"
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:37:18 GMT"
Uploaded  76c429b63adb pingpong.sif
Uploading 4a69576e405f application/vnd.oci.image.manifest.v1+json
time=2023-04-10T16:37:19-07:00 level=debug msg=Request #10
> Request URL: "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/manifests/5322c16f"
> Request method: "PUT"
> Request headers:
   "Content-Type": "application/vnd.oci.image.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.0.0"
time=2023-04-10T16:37:19-07:00 level=debug msg=Response #10
< Response Status: "201 Created"
< Response headers:
   "Docker-Content-Digest": "sha256:4a69576e405f6839bf49b7e113c035d0f9a11e40746112cf02a78dda77742274"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Location": "https://registry.gitlab.com/v2/rwaldon/pingpong/pingpong/manifests/sha256:4a69576e405f6839bf49b7e113c035d0f9a11e40746112cf02a78dda77742274"
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 10 Apr 2023 23:37:19 GMT"
   "Content-Length": "0"
Uploaded  4a69576e405f application/vnd.oci.image.manifest.v1+json
Pushed [registry] registry.gitlab.com/rwaldon/pingpong/pingpong:5322c16f
Digest: sha256:4a69576e405f[68](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L68)39bf49b7e113c035d0f9a11e40[74](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L74)6112cf02a78dda[77](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L77)74[227](https://gitlab.com/rwaldon/pingpong/-/jobs/4089508363#L227)4
Cleaning up project directory and file based variables
00:00
Job succeeded
qweeah commented 1 year ago

oras won't set limitation for uploading, the best effort oras can do is to parallelize the uploading.

time=2023-04-10T16:23:23-07:00 level=debug msg=Request #9 time=2023-04-10T16:37:19-07:00 level=debug msg=Response #9

Looks like uploading the blob sha256:76c429b63adb164a2733ee7c7ffd2ebbd6b26a490e291286fcf8fae381af8fe1 takes ~14mins. Since this is the only blob needed to be uploaded for the .sif file, there is no space to optimize the upload time on oras side.

Compressing the file before uploading may help, but looks like .sif is already compressed.

rachelwaldon commented 1 year ago

And by parallelize the uploading, you mean I could use the --concurrency flag for the push command?

rachelwaldon commented 1 year ago

Is it possible to push a container with a different tool, say docker or podman and have oras convert it to a sif on pull?

qweeah commented 1 year ago

@rachelwaldon

And by parallelize the uploading, you mean I could use the --concurrency flag for the push command?

Yes, but in my humble opinion, there is only one blob for SIF so setting --concurrency won't help in this case.

Is it possible to push a container with a different tool, say docker or podman and have oras convert it to a sif on pull?

I am not familiar with SIF eco-system and not aware of the existence of such tooling.

shizhMSFT commented 1 year ago

Slow push is network bounded. Is the GitLab service healthy at that time?

shizhMSFT commented 1 year ago

Marking this issue as a duplicate of #989