search

snowdrop / java-buildpack-client

A java buildpack (https://buildpacks.io/) platform implementation
Apache License 2.0
14 stars 7 forks source link

Docker needs authenticating to avoid unauthenticated rate limit. #95

Open cmoulliard opened 4 days ago

cmoulliard commented 4 days ago

Issue

If we execute the code of the Build me sample on a fedora VM running podman 5.2.3 with the sock file, then we got the docker limit error even if we set the username, password of the dockerclient

        String REGISTRY_USERNAME = System.getenv("REGISTRY_USERNAME");
        String REGISTRY_PASSWORD = System.getenv("REGISTRY_PASSWORD");
        String REGISTRY_SERVER = System.getenv("REGISTRY_SERVER");
        ...
        DockerClient client = getDockerClient();
        client.authConfig()
            .withUsername(REGISTRY_USERNAME)
            .withPassword(REGISTRY_PASSWORD)
            .withRegistryAddress(REGISTRY_SERVER);

        int exitCode = BuildConfig.builder()
            .withBuilderImage(new ImageReference("paketocommunity/builder-ubi-base:latest"))
            .withOutputImage(new ImageReference(IMAGE_REF))
            .withNewPlatformConfig()
              .withEnvironment(envMap)
            .endPlatformConfig()
            .withNewDockerConfig()
              .withDockerClient(client)
            .endDockerConfig()
            .withNewLogConfig()
              .withLogger(new SystemLogger())
              .withLogLevel("debug")
            .and()
            .addNewFileContentApplication(new File(PROJECT_PATH))
            .build()
            .getExitCode();

        System.exit(exitCode);

Error

[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Detector - Detect container complete, with exit code 0
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Requested Images [paketocommunity/run-java-21-ubi-base]
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Known Image : docker.io/kindest/node@sha256:047357ac0cfea04663786a612ba1eaba9702bef25227a794b52890dd8bcd692e
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Known Image : docker.io/kindest/node:v1.30.3
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Known Image : docker.io/paketocommunity/builder-ubi-base:latest
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Known Image : docker.io/paketocommunity/run-java-21-ubi-base:latest
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - pulling 'paketocommunity/run-java-21-ubi-base'
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - waiting on image paketocommunity/run-java-21-ubi-base for 60 seconds
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - success for image paketocommunity/run-java-21-ubi-base
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.docker.ImageUtils - Retrying (1) for []
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-build-urazsvvpdy at /cache-dir
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-launch-hniyhjelnr at /launch-cache-dir
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-kaniko-wcdtcpyzbu at /kaniko
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-app-njfjmjxhik at /workspace
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-platform-gnauqlwlfz at /platform
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted /var/run/user/1000/podman/podman.sock at /var/run/docker.sock
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - mounted buildpack-output-dcztckevep at /layers
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - container id b2355d8c2dd4ed9d8745482e86630775a545348e5193a2788de5b4db3b451030
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - image reference 5725dc640dbe2f9b5742825c9a2fe952ee324a8f180bc3d1bb2309a419b7d83c
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Restorer - Restorer container id b2355d8c2dd4ed9d8745482e86630775a545348e5193a2788de5b4db3b451030 will be run with uid 1002
[dev.snowdrop.BuildMe.main()] DEBUG dev.snowdrop.buildpack.lifecycle.phases.Restorer - - container args [/cnb/lifecycle/restorer, -uid, 1002, -gid, 1000, -layers, /layers, -cache-dir, /cache-dir, -log-level, debug, -build-image, paketocommunity/builder-ubi-base:latest]
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Restorer - - launching restorer container
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Restorer - - attaching log relay
2024-11-20T15:53:06Z Starting restorer...
2024-11-20T15:53:06Z Parsing inputs...
2024-11-20T15:53:06Z Ensuring privileges...
2024-11-20T15:53:06Z Executing command...
2024-11-20T15:53:06Z Pulling builder image metadata for paketocommunity/builder-ubi-base:latest...
2024-11-20T15:53:06Z ERROR: failed to pull builder image paketocommunity/builder-ubi-base:latest: failed to initialize remote image: connect to repo store "paketocommunity/builder-ubi-base:latest": GET https://index.docker.io/v2/paketocommunity/builder-ubi-base/manifests/latest: TOOMANYREQUESTS: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.phases.Restorer - Restorer container complete, with exit code 1
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - Post Build cleanup activites:
[dev.snowdrop.BuildMe.main()] INFO dev.snowdrop.buildpack.lifecycle.LifecyclePhaseFactory - - temporary build volumes removed
cmoulliard commented 4 days ago

I think that we should find a way to mount a volume containing the docker config.json or podman auth.json to the lifecycle container we are creating

cmoulliard commented 4 days ago

This old hack is still working - https://github.com/buildpacks/spec/issues/277#issuecomment-992900561

export CNB_REGISTRY_AUTH='{"index.docker.io":"Basic Y2g....3"}'

// Within the code, add the following env var
envMap.put("CNB_REGISTRY_AUTH",System.getenv("CNB_REGISTRY_AUTH"));
BarDweller commented 3 days ago

I think handling authentication in general is something the library could do better in a future revision.

Glad to see that the env var worked.. feels like we should add a section on auth to the docs to cover creation of dockerclient & setting env var appropriately, covering the scenarios where various registries need auth from different places.

For as a future feature, we could look at adding json to the ephemeral build container, that would likely be the cleanest way to provide creds to the lifecycle.

BarDweller commented 3 days ago

Also, to avoid misunderstanding here..

If we execute the code of the Build me sample on a fedora VM running podman 5.2.3 with the sock file, then we got the docker limit error

You received the docker limit error because you have exceeded dockerhub's limits for unauthenticated usage.. the same will not hold true for every user.

Executing the code of the build me sample does indeed use dockerhub, but the limits are generous enough that using the samples alone is not enough to trigger that limit.

The github action running the tests for this project also runs authenticated and has no issue.

I have altered the title of this issue to be clearer.

cmoulliard commented 3 days ago

For as a future feature, we could look at adding json to the ephemeral build container, that would likely be the cleanest way to provide creds to the lifecycle.

This is a better option and also what we do using the Buildpack Tekton's task where we mount such recredentials (= docker.json, etc) from a volume