search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
808 stars 261 forks source link

Dockerhub pull request limits - additional registry/docs #4046

Closed planetf1 closed 2 years ago

planetf1 commented 4 years ago

See https://docs.docker.com/docker-hub/download-rate-limit/ & some useful analysis at https://www.openshift.com/blog/mitigate-impact-of-docker-hub-pull-request-limits

Need to consider

tbc....

planetf1 commented 4 years ago

Dockerhub is now offering exemption for qualifying open source projects -> see https://www.docker.com/blog/expanded-support-for-open-source-software-projects/ Will see if we can get this in place cc: @jmertic

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

planetf1 commented 3 years ago

During an attempt to test the 2.10 release #5211 , the following was noted when trying to setup egeria:

 Warning  Failed          19s               kubelet            Failed to pull image "docker.io/odpi/egeria:2.10": rpc error: code = Unknown desc = Error reading manifest 2.10 in docker.io/odpi/egeria: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Following https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/ we see:

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:odpi/egeria:pull" | jq -r .token)

and decoding at jwt.io :

{
  "access": [
    {
      "type": "repository",
      "name": "odpi/egeria",
      "actions": [
        "pull"
      ],
      "parameters": {
        "pull_limit": "100",
        "pull_limit_interval": "21600"
      }
    }
  ],
  "aud": "registry.docker.io",
  "exp": 1622154518,
  "iat": 1622154218,
  "iss": "auth.docker.io",
  "jti": "fTZgVmovxlHuKuoo9K1Y",
  "nbf": 1622153918,
  "sub": ""
}
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/odpi/egeria/manifests/2.10 2>&1
HTTP/1.1 200 OK
content-length: 19568
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:e3c61ab251ec404b05d0dc92b136a7823b245690989e1b20badf30c7c6d0d792
docker-distribution-api-version: registry/2.0
etag: "sha256:e3c61ab251ec404b05d0dc92b136a7823b245690989e1b20badf30c7c6d0d792"
date: Thu, 27 May 2021 22:25:35 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 100;w=21600

This doesn't seem to indicate an issue..... as our limit is 100, with a limit remaining of 100

planetf1 commented 3 years ago

It looks as if the image limits are now controlled by IP address - so this limit may only apply to my k8s cluster - hence the odd results above (initiated from laptop)

Or the entire cloud provider...

If so we may have less of a problem. See https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ . However at this time quay.io appears to remain free/unlimited for public repos

planetf1 commented 3 years ago

dockerhub is useful for 'visibility', so we may be able to publish to both, and have our charts pointing to quay.io.

planetf1 commented 3 years ago

Even with a free account, these limits are slightly higher if authenticated when doing a pull. Additionally, rather than have the potential for sharing any free quotas with other users appearing at same IP (which may depend on network topology/cloud provider) this uses a unique user id for 'me'.

Additionally if an org is making extensive use of images they can either build locally, or pull the image into their local container registry.

As an example, for openshift by following https://docs.openshift.com/container-platform/4.7/openshift_images/managing_images/using-image-pull-secrets.html

oc create secret docker-registry dockerhub --docker-server=docker.io --docker-username=MYUSERID --docker-password='MYPASSWORD --docker-email=MYEMAIL
oc secrets link default  dockerhub --for=pull

(For odpi-egeria-lab we currently use the 'default' serviceaccount - the egeria-lab chart is different)

With this done images would then load & containers start

planetf1 commented 3 years ago

May not be request-limiting, but noticing more occurances of outages connecting to dockerhub

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

planetf1 commented 3 years ago

This is affecting operator dev at times, AND this will impact the Dojo as we will be asking many users to retrieve the containers. It needs fixing ASAP.... cc: @davidradl

planetf1 commented 3 years ago

Build process for egeria now being updated to publish to both dockerhub and quay.io We know we have users pulling from docker, plus it's the most well known registry, so for now we will continue to publish to both

References in the helm charts in egeria-charts will be updated as the container images are available on quay.io

planetf1 commented 3 years ago

Just waiting on sas-viya fix to be merged (approval required), other images done. Charts updated

planetf1 commented 3 years ago

Verified our charts.

However of note is that 'nginx' (which we use for the egeria UI, and in future may consolidate more UI access under) fails ie:

  Warning  Failed          56s                kubelet            Failed to pull image "docker.io/nginx": rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/nginx: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Not yet clear which nginx image on quay.io is appropriate.

planetf1 commented 3 years ago

Changes to our images are now almost done,

However the same issue affects any third party containers we use. WIll close the main issue and track these on a case by case basis:

planetf1 commented 3 years ago

Response from nginx team - their images are also published to the non-limited container registry: https://gallery.ecr.aws/nginx/nginx

Therefore I'm going to update our references to go here instead of dockerhub

planetf1 commented 3 years ago

Changes made in charts (merged) and egeria-ui (PR open: cc: @lpalashevski @sarbull )

Note public amazon registry is public.ecr.aws