search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
4.16k stars 446 forks source link

[BUG]: sagemaker_orchestration example gives Docker error: EOF #1580

Closed mushfiqulIslam closed 1 year ago

mushfiqulIslam commented 1 year ago

Contact Details [Optional]

No response

System Information

ZENML_LOCAL_VERSION: 0.40.1 ZENML_SERVER_VERSION: 0.40.1 ZENML_SERVER_DATABASE: sqlite ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /root/.config/zenml ZENML_LOCAL_STORE_DIR: /root/.config/zenml/local_stores ZENML_SERVER_URL: http://0.0.0.0:8237 ZENML_ACTIVE_REPOSITORY_ROOT: /home/mushfiq/zenml_examples/sagemaker_orchestration PYTHON_VERSION: 3.10.6 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'} ACTIVE_WORKSPACE: default ACTIVE_STACK: test_sagemaker_stack ACTIVE_USER: mushfiq TELEMETRY_STATUS: enabled

INTEGRATIONS: ['aws', 'kaniko', 'mlflow', 'pillow', 'pytorch', 's3', 'scipy', 'sklearn', 'tensorboard', 'tensorflow'] PACKAGES: {'s3fs': '2023.3.0', 'fsspec': '2023.3.0', 'certifi': '2022.12.7', 'pytz': '2022.6', 'setuptools': '59.6.0', 'cryptography': '38.0.4', 'kubernetes': '26.1.0', 'pyzmq': '24.0.1', 'flatbuffers': '22.12.6', 'gevent': '22.10.2', 'aiofiles': '22.1.0', 'pip': '22.0.2', 'packaging': '22.0', 'contextlib2': '21.6.0', 'argon2-cffi': '21.3.0', 'argon2-cffi-bindings': '21.2.0', 'isoduration': '20.11.0', 'attrs': '20.3.0', 'gunicorn': '20.1.0', 'lit': '16.0.5', 'libclang': '14.0.6', 'rich': '12.6.0', 'nvidia-cublas-cu11': '11.10.3.66', 'nvidia-cuda-cupti-cu11': '11.7.101', 'nvidia-cuda-runtime-cu11': '11.7.99', 'nvidia-cuda-nvrtc-cu11': '11.7.99', 'nvidia-nvtx-cu11': '11.7.91', 'nvidia-cusparse-cu11': '11.7.4.91', 'nvidia-cusolver-cu11': '11.4.0.1', 'nvidia-cufft-cu11': '10.9.0.58', 'websockets': '10.4', 'nvidia-curand-cu11': '10.2.10.91', 'pillow': '9.5.0', 'environs': '9.5.0', 'ipython': '8.7.0', 'nvidia-cudnn-cu11': '8.5.0.96', 'click': '8.1.3', 'tenacity': '8.1.0', 'ipywidgets': '7.7.2', 'jupyter-client': '7.4.8', 'nbconvert': '7.2.6', 'ipykernel': '6.19.0', 'notebook': '6.5.2', 'tornado': '6.2', 'docker': '6.1.2', 'multidict': '6.0.3', 'plotly': '5.11.0', 'importlib-resources': '5.10.1', 'psutil': '5.9.4', 'nbformat': '5.7.0', 'traitlets': '5.6.0', 'zope.interface': '5.5.2', 'pyyaml': '5.4.1', 'cachetools': '5.2.0', 'decorator': '5.1.1', 'jupyter-core': '5.1.0', 'bleach': '5.0.1', 'smmap': '5.0.0', 'jsonschema': '4.17.3', 'importlib-metadata': '4.13.0', 'beautifulsoup4': '4.11.1', 'rsa': '4.9', 'pexpect': '4.8.0', 'zope.event': '4.5.0', 'typing-extensions': '4.4.0', 'oauth2client': '4.1.3', 'gitdb': '4.0.10', 'async-timeout': '4.0.2', 'bcrypt': '4.0.1', 'cmake': '3.26.3', 'protobuf': '3.19.6', 'marshmallow': '3.19.0', 'filelock': '3.12.0', 'zipp': '3.11.0', 'orjson': '3.8.3', 'aiohttp': '3.8.3', 'h5py': '3.7.0', 'anyio': '3.6.2', 'widgetsnbextension': '3.6.1', 'asgiref': '3.5.2', 'markdown': '3.4.1', 'idna': '3.4', 'python-jose': '3.3.0', 'opt-einsum': '3.3.0', 'oauthlib': '3.2.2', 'gitpython': '3.1.29', 'jinja2': '3.1.2', 'threadpoolctl': '3.1.0', 'networkx': '3.1', 'prompt-toolkit': '3.0.36', 'flask-sqlalchemy': '3.0.3', 'uritemplate': '3.0.1', 'sagemaker': '2.117.0', 'requests': '2.28.1', 'tritonclient': '2.28.0', 'pycparser': '2.21', 'fastjsonschema': '2.16.2', 'google-auth': '2.15.0', 'nvidia-nccl-cu11': '2.14.3', 'pygments': '2.13.0', 'google-api-core': '2.11.0', 'psycopg2-binary': '2.9.5', 'python-dateutil': '2.8.2', 'tensorflow': '2.8.0', 'keras': '2.8.0', 'tensorflow-estimator': '2.8.0', 'tensorboard': '2.8.0', 'tf-estimator-nightly': '2.8.0.dev2021122109', 'pyjwt': '2.6.0', 'platformdirs': '2.6.0', 'aiobotocore': '2.4.2', 'pyparsing': '2.4.2', 'soupsieve': '2.3.2.post1', 'jsonpointer': '2.3', 'flask': '2.2.2', 'werkzeug': '2.2.2', 'asttokens': '2.2.1', 'cloudpickle': '2.2.0', 'itsdangerous': '2.1.2', 'charset-normalizer': '2.1.1', 'termcolor': '2.1.1', 'markupsafe': '2.1.1', 'python-json-logger': '2.0.4', 'mistune': '2.0.4', 'geventhttpclient': '2.0.2', 'kafka-python': '2.0.2', 'torch': '2.0.1', 'greenlet': '2.0.1', 'triton': '2.0.0', 'jupyter-server': '2.0.0', 'googleapis-common-protos': '1.57.0', 'grpcio': '1.51.1', 'botocore': '1.27.59', 'urllib3': '1.26.13', 'mlflow': '1.25.1', 'boto3': '1.24.59', 'numpy': '1.23.5', 'six': '1.16.0', 'cffi': '1.15.1', 'wrapt': '1.14.1', 'google-api-python-client': '1.12.11', 'webcolors': '1.12', 'sympy': '1.12', 'backoff': '1.10.0', 'scipy': '1.9.3', 'pydantic': '1.9.2', 'python-rapidjson': '1.9', 'yarl': '1.8.2', 'alembic': '1.8.1', 'tensorboard-plugin-wit': '1.8.1', 'distro': '1.8.0', 'send2trash': '1.8.0', 'ppft': '1.7.6.6', 'passlib': '1.7.4', 'debugpy': '1.6.4', 'astunparse': '1.6.3', 'monotonic': '1.6', 'nest-asyncio': '1.5.6', 'portpicker': '1.5.2', 'pandas': '1.5.2', 'fqdn': '1.5.1', 'pandocfilters': '1.5.0', 'sqlalchemy': '1.4.41', 'websocket-client': '1.4.2', 'analytics-python': '1.4.0', 'pkgutil-resolve-name': '1.3.10', 'frozenlist': '1.3.3', 'aiosignal': '1.3.1', 'requests-oauthlib': '1.3.1', 'mpmath': '1.3.0', 'sniffio': '1.3.0', 'absl-py': '1.3.0', 'querystring-parser': '1.2.4', 'mako': '1.2.4', 'arrow': '1.2.3', 'tinycss2': '1.2.1', 'executing': '1.2.0', 'uri-template': '1.2.0', 'mlserver-mlflow': '1.2.0', 'joblib': '1.2.0', 'mlserver': '1.2.0', 'scikit-learn': '1.1.3', 'keras-preprocessing': '1.1.2', 'jupyterlab-widgets': '1.1.1', 'brotli': '1.0.9', 'pymysql': '1.0.2', 'jmespath': '1.0.1', 'smdebug-rulesconfig': '1.0.1', 'fastapi': '0.75.2', 'multiprocess': '0.70.14', 'zenml': '0.40.1', 'wheel': '0.38.4', 'sqlalchemy-utils': '0.38.3', 'tensorflow-io-gcs-filesystem': '0.24.0', 'tensorflow-io': '0.24.0', 'prometheus-flask-exporter': '0.21.0', 'python-dotenv': '0.21.0', 'pyrsistent': '0.19.2', 'httplib2': '0.19.1', 'validators': '0.18.2', 'jedi': '0.18.2', 'fasteners': '0.18', 'ecdsa': '0.18.0', 'uvicorn': '0.17.6', 'databricks-cli': '0.17.4', 'starlette': '0.17.1', 'terminado': '0.17.1', 'uvloop': '0.17.0', 'torchvision': '0.15.2', 'prometheus-client': '0.15.0', 'h11': '0.14.0', 'starlette-exporter': '0.14.0', 'statsmodels': '0.13.5', 'ml-metadata': '0.13.1.dev0', 'aioitertools': '0.11.0', 'python-terraform': '0.10.1', 'commonmark': '0.9.1', 'publicsuffixlist': '0.9.1', 'tabulate': '0.9.0', 'parso': '0.8.3', 'watchgod': '0.8.2', 'aiokafka': '0.8.0', 'schema': '0.7.5', 'pickleshare': '0.7.5', 'nbclient': '0.7.2', 'defusedxml': '0.7.1', 'ptyprocess': '0.7.0', 'py-grpc-prometheus': '0.7.0', 'stack-data': '0.6.2', 's3transfer': '0.6.1', 'tensorboard-data-server': '0.6.1', 'dataclasses': '0.6', 'google-apitools': '0.5.32', 'patsy': '0.5.3', 'webencodings': '0.5.1', 'httptools': '0.5.0', 'jupyter-events': '0.5.0', 'nbclassic': '0.4.8', 'pyasn1': '0.4.8', 'google-auth-oauthlib': '0.4.6', 'sqlparse': '0.4.3', 'jupyter-server-terminals': '0.4.2', 'gast': '0.4.0', 'entrypoints': '0.4', 'dill': '0.3.6', 'pox': '0.3.2', 'click-params': '0.3.0', 'pathos': '0.3.0', 'pyasn1-modules': '0.2.8', 'wcwidth': '0.2.5', 'notebook-shim': '0.2.2', 'jupyterlab-pygments': '0.2.2', 'pure-eval': '0.2.2', 'fastapi-utils': '0.2.1', 'ipython-genutils': '0.2.0', 'backcall': '0.2.0', 'google-pasta': '0.2.0', 'evidently': '0.1.52.dev0', 'matplotlib-inline': '0.1.6', 'protobuf3-to-dict': '0.1.5', 'rfc3339-validator': '0.1.4', 'comm': '0.1.1', 'rfc3986-validator': '0.1.1', 'google-auth-httplib2': '0.1.0', 'sqlmodel': '0.0.8', 'python-multipart': '0.0.5', 'sqlalchemy2-stubs': '0.0.2a29'}

What happened?

I ran sagemaker_orchestration according to provided guideline: https://github.com/zenml-io/zenml/tree/0.40.1/examples/sagemaker_orchestration. While pushing docker image to sagemaker EOF docker error raised.

Reproduction steps

Steps given here: https://github.com/zenml-io/zenml/tree/0.40.1/examples/sagemaker_orchestration

Relevant log output

Reusing registered pipeline fashion_mnist_pipeline (version: 1).
Building Docker image(s) for pipeline fashion_mnist_pipeline.
Building Docker image 123456789876.dkr.ecr.us-east-1.amazonaws.com/zenml:fashion_mnist_pipeline-orchestrator.
- Including user-defined requirements: torchvision
- Including integration requirements: boto3<=1.24.59, s3fs>2022.3.0,<=2023.4.0, sagemaker==2.117.0, torch
No .dockerignore found, including all files inside build context.
Build context size for docker image: 59.23 MiB. If you believe this is unreasonably large, make sure to include a .dockerignore file at the root of your build context /home/mushfiq/zenml_examples/sagemaker_orchestration/.dockerignore or specify a custom file in the Docker configuration when defining your pipeline.
Step 1/10 : FROM zenmldocker/zenml:0.40.1-py3.10
Step 2/10 : WORKDIR /app
Step 3/10 : COPY .zenml_user_requirements .
Step 4/10 : RUN pip install --default-timeout=60 --no-cache-dir  -r .zenml_user_requirements
Step 5/10 : COPY .zenml_integration_requirements .
Step 6/10 : RUN pip install --default-timeout=60 --no-cache-dir  -r .zenml_integration_requirements
Step 7/10 : ENV ZENML_ENABLE_REPO_INIT_WARNINGS=False
Step 8/10 : ENV ZENML_CONFIG_PATH=/app/.zenconfig
Step 9/10 : COPY . .
Step 10/10 : RUN chmod -R a+rw .
Pushing Docker image 123456789876.dkr.ecr.us-east-1.amazonaws.com/zenml:fashion_mnist_pipeline-orchestrator.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/mushfiq/zenml_examples/sagemaker_orchestration/run.py:18 in <module>                       │
│                                                                                                  │
│   15 from pipelines.fashion_mnist_pipeline import fashion_mnist_pipeline                         │
│   16                                                                                             │
│   17 if __name__ == "__main__":                                                                  │
│ ❱ 18 │   fashion_mnist_pipeline()                                                                │
│   19                                                                                             │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/new/pipelines/pipeline.py:1167 in    │
│ __call__                                                                                         │
│                                                                                                  │
│   1164 │   │   │   return self.entrypoint(*args, **kwargs)                                       │
│   1165 │   │                                                                                     │
│   1166 │   │   self.prepare(*args, **kwargs)                                                     │
│ ❱ 1167 │   │   return self._run(**self._run_args)                                                │
│   1168 │                                                                                         │
│   1169 │   def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None:                        │
│   1170 │   │   """Calls the pipeline entrypoint function with the given arguments.               │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/new/pipelines/pipeline.py:597 in     │
│ _run                                                                                             │
│                                                                                                  │
│    594 │   │   │   │   deployment=deployment, local_repo_context=local_repo_context              │
│    595 │   │   │   )                                                                             │
│    596 │   │   │                                                                                 │
│ ❱  597 │   │   │   build_model = build_utils.reuse_or_create_pipeline_build(                     │
│    598 │   │   │   │   deployment=deployment,                                                    │
│    599 │   │   │   │   pipeline_id=pipeline_id,                                                  │
│    600 │   │   │   │   allow_build_reuse=not prevent_build_reuse,                                │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/new/pipelines/build_utils.py:96 in   │
│ reuse_or_create_pipeline_build                                                                   │
│                                                                                                  │
│    93 │   │   │   │   )                                                                          │
│    94 │   │   │   │   return existing_build                                                      │
│    95 │   │                                                                                      │
│ ❱  96 │   │   return create_pipeline_build(                                                      │
│    97 │   │   │   deployment=deployment,                                                         │
│    98 │   │   │   pipeline_id=pipeline_id,                                                       │
│    99 │   │   │   code_repository=code_repository,                                               │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/new/pipelines/build_utils.py:261 in  │
│ create_pipeline_build                                                                            │
│                                                                                                  │
│   258 │   │   │   │   image_name_or_digest,                                                      │
│   259 │   │   │   │   dockerfile,                                                                │
│   260 │   │   │   │   requirements,                                                              │
│ ❱ 261 │   │   │   ) = docker_image_builder.build_docker_image(                                   │
│   262 │   │   │   │   docker_settings=build_config.settings,                                     │
│   263 │   │   │   │   tag=tag,                                                                   │
│   264 │   │   │   │   stack=stack,                                                               │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/utils/pipeline_docker_image_builder. │
│ py:302 in build_docker_image                                                                     │
│                                                                                                  │
│   299 │   │   │   │   │   │   destination=destination, source=source                             │
│   300 │   │   │   │   │   )                                                                      │
│   301 │   │   │                                                                                  │
│ ❱ 302 │   │   │   image_name_or_digest = image_builder.build(                                    │
│   303 │   │   │   │   image_name=target_image_name,                                              │
│   304 │   │   │   │   build_context=build_context,                                               │
│   305 │   │   │   │   docker_build_options=build_options,                                        │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/image_builders/local_image_builder.p │
│ y:119 in build                                                                                   │
│                                                                                                  │
│   116 │   │   docker_utils._process_stream(output_stream)                                        │
│   117 │   │                                                                                      │
│   118 │   │   if container_registry:                                                             │
│ ❱ 119 │   │   │   return container_registry.push_image(image_name)                               │
│   120 │   │   else:                                                                              │
│   121 │   │   │   return image_name                                                              │
│   122                                                                                            │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/container_registries/base_container_ │
│ registry.py:180 in push_image                                                                    │
│                                                                                                  │
│   177 │   │   │   )                                                                              │
│   178 │   │                                                                                      │
│   179 │   │   self.prepare_image_push(image_name)                                                │
│ ❱ 180 │   │   return docker_utils.push_image(                                                    │
│   181 │   │   │   image_name, docker_client=self.docker_client                                   │
│   182 │   │   )                                                                                  │
│   183                                                                                            │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/utils/docker_utils.py:263 in         │
│ push_image                                                                                       │
│                                                                                                  │
│   260 │   logger.info("Pushing Docker image `%s`.", image_name)                                  │
│   261 │   docker_client = docker_client or DockerClient.from_env()                               │
│   262 │   output_stream = docker_client.images.push(image_name, stream=True)                     │
│ ❱ 263 │   aux_info = _process_stream(output_stream)                                              │
│   264 │   logger.info("Finished pushing Docker image.")                                          │
│   265 │                                                                                          │
│   266 │   image_name_without_tag, _ = image_name.rsplit(":", maxsplit=1)                         │
│                                                                                                  │
│ /opt/template/venv/lib/python3.10/site-packages/zenml/utils/docker_utils.py:357 in         │
│ _process_stream                                                                                  │
│                                                                                                  │
│   354 │   │   │   try:                                                                           │
│   355 │   │   │   │   line_json = json.loads(line)                                               │
│   356 │   │   │   │   if "error" in line_json:                                                   │
│ ❱ 357 │   │   │   │   │   raise RuntimeError(f"Docker error: {line_json['error']}.")             │
│   358 │   │   │   │   elif "stream" in line_json:                                                │
│   359 │   │   │   │   │   text = line_json["stream"].strip()                                     │
│   360 │   │   │   │   │   if "ERROR" in text:                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Docker error: EOF.

Code of Conduct

htahir1 commented 1 year ago

@mushfiqulIslam Does your remote repository exist? I see here that this error might be caused if you try to push to a repository that does not exist in AWS

mushfiqulIslam commented 1 year ago

yes, a repository with name"zenml" was previously created and available. My local machines docker version: 23.0.6

htahir1 commented 1 year ago

@mushfiqulIslam Just to confirm we’re talking about the docker repository

mushfiqulIslam commented 1 year ago

@htahir1, I have created an aws ecr repository but did not create any repository on docker hub.

htahir1 commented 1 year ago

Does this exist on your AWS Docker registry? 123456789876.dkr.ecr.us-east-1.amazonaws.com/zenml

mushfiqulIslam commented 1 year ago

The provided 123456789876.dkr.ecr.us-east-1.amazonaws.com/zenml is dummy. Exact one is available in AWS Docker registry and visibility is private.

htahir1 commented 1 year ago

@mushfiqulIslam Then my guess is that either that private repository does not exist or you do have permission locally to push to it

mushfiqulIslam commented 1 year ago

Found the access related issue. Basically, AWS EC2 Container registry full access was needed to upload docker image.