uber / kraken

P2P Docker registry capable of distributing TBs of data in seconds
Apache License 2.0
6.14k stars 423 forks source link

500 failed to verify data: digests do not match; Image fails to pull. #303

Open will-ds opened 3 years ago

will-ds commented 3 years ago

Describe the bug Pulling an image (whether it be the origin, agent or manually) returns the following in the origin logs (/var/log/kraken/kraken-origin/) after re-deploying kraken. Image fails to pull.

INFO    handler/handler.go:98   500 GET /namespace/nvidia/cuda/blobs/sha256:2806d80fe28f9a42dfe4a1b30aeaefb3f75436d3c7fbfc6dfe7e71e1e5d3840a failed to verify data: digests do not match

To Reproduce Steps to reproduce the behavior:

  1. Spin up Kraken using the helm chart. Configure GCS as the backend.
  2. Push an image via the kraken proxy.
  3. Run a separate deployment that uses this image. Verify it gets pulled and runs correctly.
  4. Tear down both deployments.
  5. Re-deploy kraken using helm chart using same configuration as Step 1.
  6. Run a new deployment using the same yaml file as Step 3.

Check logs of kraken-origin to find the above error.

Expected behavior Image should pull down to origin(s).

Screenshots If applicable, add screenshots to help explain your problem.

Environments

Additional context Add any other context about the problem here--environment configurations, logs, etc.

Build index backend configuration:

    - namespace: .*
      backend:
        gcs:
          username: kraken-service-account
          bucket: gke_image_cache
          root_directory: /tags/
          name_path: docker_tag

Oirigin backend configuration:

    - namespace: .*
      backend:
        gcs:
          username: kraken-service-account
          bucket: gke_image_cache
          root_directory: /blobs/
          name_path: identity
ish-xyz commented 3 years ago

@will-ds I have the same error, did you find any solution? Unfortunately, debug logs don't return any additional info.