[BUG] mlflow build-and-push-container doesn't work on sagemaker

mohitanchlia commented 1 year ago

Issues Policy acknowledgement

[X] I have read and agree to submit bug reports in accordance with the issues policy

Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

MLflow version

Client: 2.3
Tracking server: 2.3

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Darwin MM7RQJXW7Q 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64
Python version: 3.8 and 3.11
yarn version, if running the dev UI:

Describe the problem

[BUG] mlflow build-and-push-container doesn't work on sagemaker. The command completes successfully and it registers the image on the container. However, when I run deployment it fails to bring up the sagemaker instance. In Sagemaker conatiner log I see the following error:

`Cloudwatch logs from sagemaker

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 57, in _init _serve(env_manager) File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 84, in _serve _serve_pyfunc(m, env_manager) File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 156, in _serve_pyfunc _install_pyfunc_deps( File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 120, in _install_pyfunc_deps env_activate_cmd = _get_or_create_virtualenv(model_path) File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/virtualenv.py", line 364, in _get_or_create_virtualenv python_bin_path = _install_python( File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/virtualenv.py", line 133, in _install_python _exec_cmd( File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/process.py", line 117, in _exec_cmd raise ShellCommandException.from_completed_process(comp_process) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 57, in _init _serve(env_manager) File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 84, in _serve _serve_pyfunc(m, env_manager) File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 156, in _serve_pyfunc _install_pyfunc_deps( File "/usr/local/lib/python3.8/dist-packages/mlflow/models/container/init.py", line 120, in _install_pyfunc_deps env_activate_cmd = _get_or_create_virtualenv(model_path) File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/virtualenv.py", line 364, in _get_or_create_virtualenv python_bin_path = _install_python( File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/virtualenv.py", line 133, in _install_python _exec_cmd( File "/usr/local/lib/python3.8/dist-packages/mlflow/utils/process.py", line 117, in _exec_cmd raise ShellCommandException.from_completed_process(comp_process)`

When I run this image locally with just run command and no args I still get File "<string>", line 1, in <module>

I am stuck at this point as there is no visibility in this

Tracking information

REPLACE_ME

Code to reproduce issue

REPLACE_ME

Stack trace

REPLACE_ME

Other info / logs

REPLACE_ME

What component(s) does this bug affect?

[ ] area/artifacts: Artifact stores and artifact logging
[ ] area/build: Build and test infrastructure for MLflow
[ ] area/docs: MLflow documentation pages
[ ] area/examples: Example code
[ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
[ ] area/models: MLmodel format, model serialization/deserialization, flavors
[ ] area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
[ ] area/projects: MLproject format, project running backends
[ ] area/scoring: MLflow Model server, model deployment tools, Spark UDFs
[X] area/server-infra: MLflow Tracking server backend
[ ] area/tracking: Tracking Service, tracking client APIs, autologging

What interface(s) does this bug affect?

[ ] area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
[X] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
[ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
[ ] area/windows: Windows support

What language(s) does this bug affect?

[ ] language/r: R APIs and clients
[ ] language/java: Java APIs and clients
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[ ] integrations/azure: Azure and Azure ML integrations
[X] integrations/sagemaker: SageMaker integrations
[ ] integrations/databricks: Databricks integrations

BenWilson2 commented 1 year ago

Can you share a repro?

BenWilson2 commented 1 year ago

Can you try this with MLflow 1.30.x and see if this issue still presents itself? Thank you!

mlflow-automation commented 1 year ago

@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.

mohitanchlia commented 1 year ago

Can you try this with MLflow 1.30.x and see if this issue still presents itself? Thank you!

I am using mlflow 1.32 I believe

mohitanchlia commented 1 year ago

Can you share a repro?

https://github.com/aws-samples/amazon-sagemaker-mlflow-fargate/tree/main

pdifranc commented 1 year ago

Can you share a repro?

https://github.com/aws-samples/amazon-sagemaker-mlflow-fargate/tree/main

This repo deploys MLflow 2.0.1. There has been a major change in the mlflow sdk on how to deploy to sagemaker. Can you please make sure your MLflow SDK, MLflow tracking server are the same?

mohitanchlia commented 1 year ago

Can you share a repro?

https://github.com/aws-samples/amazon-sagemaker-mlflow-fargate/tree/main

This repo deploys MLflow 2.0.1. There has been a major change in the mlflow sdk on how to deploy to sagemaker. Can you please make sure your MLflow SDK, MLflow tracking server are the same?

I verified that I am using 2.0.1 client

mlflow / mlflow