Deploying a Flow with Docker Desktop in Europe raises BuildError: failed to export image: NotFound: content digest

anze3db commented 1 day ago

Bug summary

Trying to create a deployment with a Docker Image on MacOS with Docker Desktop fails with the following error, but only if you are located in Europe:

  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 399, in coroutine_wrapper
    return run_coro_as_sync(ctx_call())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 243, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 312, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 182, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/zidar/.asdf/installs/python/3.12.1/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 383, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 225, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 389, in ctx_call
    result = await async_fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/deployments/runner.py", line 925, in deploy
    image.build()
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/docker/docker_image.py", line 73, in build
    build_image(**build_kwargs)
  File "/Users/zidar/.asdf/installs/python/3.12.1/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/zidar/programming/app/.venv/lib/python3.12/site-packages/prefect/utilities/dockerutils.py", line 194, in build_image
    raise BuildError(event["error"])
prefect.utilities.dockerutils.BuildError: failed to export image: NotFound: content digest sha256:7fb66093b170bccb413f3e1c8f4b92fa440ea68fc4cddccf4c3b47e2673cfb9c: not found

If you change your location (with a VPN) to the US, the issue does not reproduce. If you use OrbStack instead of Docker Desktop, the issue also does not reproduce.

We figured this out because coworkers in US had no trouble creating the deployment, but others in the EU constantly get the error.

Example deployment code:

job.deploy(
    work_pool_name=work_pool_name,
    image=DockerImage(
        name=docker_image_name,
        platform="linux/amd64",
        dockerfile="Dockerfile",
        target=target,
    ),
)

Building the Dockerfile manually doesn't raise this error.

Version info

Version:             3.1.0
API version:         0.8.4
Python version:      3.12.6
Git commit:          a83ba39b
Built:               Thu, Oct 31, 2024 12:43 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         server
Pydantic version:    2.9.2
Integrations:
  prefect-docker:    0.6.1

Additional context

This issue has also popped up in the Prefect Community Slack: https://prefect-community.slack.com/archives/CL09KU1K7/p1730205889746789

teocns commented 11 hours ago

Just out of curiosity, does this also occur when explicitly passing the registry name in the image?

docker.io/your_username/image:tag

anze3db commented 9 hours ago

I haven't tried docker.io, but the issue reproduces with AWS ECR, no matter how I specify the registry name in the image.

The issue even reproduces if neither ECR nor docker.io configured and I'm building and using the image locally without pushing it to a remote repository.

PrefectHQ / prefect