search

Azure / azureml-examples

Official community-driven Azure Machine Learning examples, tested with GitHub Actions.
https://docs.microsoft.com/azure/machine-learning
MIT License
1.77k stars 1.44k forks source link

tutorials/azureml-getting-started (credits-default-model) fails to deploy with: "Expected 96 from C header, got 88 from PyObject" #3415

Open RoFz opened 1 month ago

RoFz commented 1 month ago

Operating System

MacOS

Version Information

Python Version: 3.12.7 azure-ai-ml package version: 1.21.0

Steps to reproduce

  1. Load azureml-getting-started-studio.ipynb into VS Code
  2. Add the code block below before the first original block (otherwise, authentication fails) 
    
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential

try: credential = DefaultAzureCredential()

Check if given credential can get token successfully.

credential.get_token("https://management.azure.com/.default")

except Exception as ex:

Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work

# This will open a browser page for
credential = InteractiveBrowserCredential()

3. Run everything successfully until step/item 7 (_Deploy the model as an online endpoint_)
4. Follow the _Endpoint creation & deployment via wizard UI_ instructions, receive a **Failed** under the deployment's _Provisioning State_ and the error log below:

Instance status: SystemSetup: Succeeded UserContainerImagePull: Succeeded ModelDownload: Succeeded UserContainerStart: InProgress

Container events: Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:12.478572Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:22.494146Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:32.498042Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:42.497885Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:03:52.478491Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:02.494329Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:12.494084Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:22.494119Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:32.494126Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:42.497857Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:04:52.494157Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:02.49442Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:12.494019Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:22.494261Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:32.494194Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:42.49788Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:05:52.494265Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:02.49823Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:12.494149Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502 Kind: Pod, Name: ReadinessProbeFailed, Type: Warning, Time: 2024-10-10T11:06:22.494347Z, Message: Readiness probe failed: HTTP probe failed with statuscode: 502

Container logs: 2024-10-10T11:00:05,636783121+00:00 - rsyslog/run 2024-10-10T11:00:05,660534689+00:00 - nginx/run 2024-10-10T11:00:05,666009973+00:00 - gunicorn/run nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1 2024-10-10T11:00:05,699105686+00:00 | gunicorn/run | 2024-10-10T11:00:05,706316197+00:00 | gunicorn/run | ############################################### 2024-10-10T11:00:05,710913868+00:00 | gunicorn/run | AzureML Container Runtime Information 2024-10-10T11:00:05,717056363+00:00 | gunicorn/run | ############################################### 2024-10-10T11:00:05,720074210+00:00 | gunicorn/run | 2024-10-10T11:00:05,723216459+00:00 | gunicorn/run | 2024-10-10T11:00:05,729737860+00:00 | gunicorn/run | AzureML image information: mlflow-ubuntu20.04-py38-cpu-inference:20240805.v5 2024-10-10T11:00:05,732576004+00:00 | gunicorn/run | 2024-10-10T11:00:05,735530049+00:00 | gunicorn/run | 2024-10-10T11:00:05,738387494+00:00 | gunicorn/run | PATH environment variable: /opt/miniconda/envs/amlenv/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-10T11:00:05,741136636+00:00 | gunicorn/run | PYTHONPATH environment variable: 2024-10-10T11:00:05,795602279+00:00 | gunicorn/run | 2024-10-10T11:00:11,635188772+00:00 | gunicorn/run | CONDAPATH environment variable: /opt/miniconda

conda environments:

# base /opt/miniconda amlenv /opt/miniconda/envs/amlenv

2024-10-10T11:00:14,920278755+00:00 | gunicorn/run | 2024-10-10T11:00:14,925375932+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)

annotated-types==0.7.0 azure-core==1.30.2 azure-identity==1.17.1 azureml-inference-server-http==1.3.0 blinker==1.8.2 cachetools==5.4.0 certifi==2024.7.4 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cryptography==43.0.0 Flask==2.3.2 Flask-Cors==3.0.10 google-api-core==2.19.1 google-auth==2.32.0 googleapis-common-protos==1.63.2 gunicorn==22.0.0 idna==3.7 importlib_metadata==8.2.0 inference-schema==1.8 itsdangerous==2.2.0 Jinja2==3.1.4 MarkupSafe==2.1.5 msal==1.30.0 msal-extensions==1.2.0 opencensus==0.11.4 opencensus-context==0.1.3 opencensus-ext-azure==1.1.13 packaging==24.1 portalocker==2.10.1 proto-plus==1.24.0 protobuf==5.27.3 psutil==6.0.0 pyasn1==0.6.0 pyasn1_modules==0.4.0 pycparser==2.22 pydantic==2.7.4 pydantic-settings==2.4.0 pydantic_core==2.18.4 PyJWT==2.9.0 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 pytz==2024.1 requests==2.32.3 rsa==4.9 six==1.16.0 typing_extensions==4.12.2 urllib3==2.2.2 Werkzeug==3.0.3 wrapt==1.16.0 zipp==3.19.2

2024-10-10T11:00:20,606892801+00:00 | gunicorn/run | 2024-10-10T11:00:20,610213452+00:00 | gunicorn/run | Entry script directory: /var/mlflow_resources/. 2024-10-10T11:00:20,613244298+00:00 | gunicorn/run | 2024-10-10T11:00:20,618317077+00:00 | gunicorn/run | ############################################### 2024-10-10T11:00:20,621522426+00:00 | gunicorn/run | Dynamic Python Package Installation 2024-10-10T11:00:20,624719075+00:00 | gunicorn/run | ############################################### 2024-10-10T11:00:20,627784922+00:00 | gunicorn/run | 2024-10-10T11:00:20,631190575+00:00 | gunicorn/run | Updating conda environment from /var/azureml-app/azureml-models/credit_defaults_model/1/credit_defaults_model/conda.yaml ! Retrieving notices: ...working... done Channels:

==> WARNING: A newer version of conda exists. <== current version: 24.7.1 latest version: 24.9.1

Please update conda by running

$ conda update -n base -c conda-forge conda

Downloading and Extracting Packages: ...working... done Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Installing pip dependencies: ...working... Ran pip subprocess with arguments: ['/opt/miniconda/envs/userenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.v_2c6wn8.requirements.txt', '--exists-action=b'] Pip subprocess output: Collecting mlflow==2.16.2 Downloading mlflow-2.16.2-py3-none-any.whl (26.7 MB) Collecting backports-tempfile==1.0 Downloading backports.tempfile-1.0-py2.py3-none-any.whl (4.4 kB) Collecting cloudpickle==2.2.1 Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB) Collecting jaraco-collections==5.1.0 Downloading jaraco.collections-5.1.0-py3-none-any.whl (11 kB) Collecting pandas==1.5.3 Downloading pandas-1.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.1 MB) Collecting psutil==5.8.0 Downloading psutil-5.8.0.tar.gz (470 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting scikit-learn==1.5.2 Downloading scikit_learn-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.3 MB) Collecting graphene<4 Downloading graphene-3.3-py2.py3-none-any.whl (128 kB) Collecting sqlalchemy<3,>=1.4.0 Downloading SQLAlchemy-2.0.35-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB) Collecting alembic!=1.10.0,<2 Downloading alembic-1.13.3-py3-none-any.whl (233 kB) Collecting markdown<4,>=3.3 Downloading Markdown-3.7-py3-none-any.whl (106 kB) Collecting matplotlib<4 Downloading matplotlib-3.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB) Collecting Flask<4 Downloading flask-3.0.3-py3-none-any.whl (101 kB) Collecting mlflow-skinny==2.16.2 Downloading mlflow_skinny-2.16.2-py3-none-any.whl (5.6 MB) Collecting numpy<3 Downloading numpy-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB) Collecting Jinja2<4,>=2.11 Downloading jinja2-3.1.4-py3-none-any.whl (133 kB) Collecting gunicorn<24 Downloading gunicorn-23.0.0-py3-none-any.whl (85 kB) Collecting pyarrow<18,>=4.0.0 Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB) Collecting docker<8,>=4.0.0 Downloading docker-7.1.0-py3-none-any.whl (147 kB) Collecting scipy<2 Downloading scipy-1.14.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.2 MB) Collecting backports.weakref Downloading backports.weakref-1.0.post1-py2.py3-none-any.whl (5.2 kB) Collecting jaraco.text Downloading jaraco.text-4.0.0-py3-none-any.whl (11 kB) Collecting python-dateutil>=2.8.1 Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting pytz>=2020.1 Downloading pytz-2024.2-py2.py3-none-any.whl (508 kB) Collecting joblib>=1.2.0 Downloading joblib-1.4.2-py3-none-any.whl (301 kB) Collecting threadpoolctl>=3.1.0 Downloading threadpoolctl-3.5.0-py3-none-any.whl (18 kB) Collecting sqlparse<1,>=0.4.0 Downloading sqlparse-0.5.1-py3-none-any.whl (44 kB) Collecting opentelemetry-api<3,>=1.9.0 Downloading opentelemetry_api-1.27.0-py3-none-any.whl (63 kB) Collecting click<9,>=7.0 Downloading click-8.1.7-py3-none-any.whl (97 kB) Collecting gitpython<4,>=3.1.9 Downloading GitPython-3.1.43-py3-none-any.whl (207 kB) Collecting importlib-metadata!=4.7.0,<9,>=3.7.0 Downloading importlib_metadata-8.5.0-py3-none-any.whl (26 kB) Collecting pyyaml<7,>=5.1 Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB) Collecting cachetools<6,>=5.0.0 Downloading cachetools-5.5.0-py3-none-any.whl (9.5 kB) Collecting databricks-sdk<1,>=0.20.0 Downloading databricks_sdk-0.34.0-py3-none-any.whl (565 kB) Collecting opentelemetry-sdk<3,>=1.9.0 Downloading opentelemetry_sdk-1.27.0-py3-none-any.whl (110 kB) Collecting requests<3,>=2.17.3 Downloading requests-2.32.3-py3-none-any.whl (64 kB) Collecting protobuf<6,>=3.12.0 Downloading protobuf-5.28.2-cp38-abi3-manylinux2014_x86_64.whl (316 kB) Collecting packaging<25 Downloading packaging-24.1-py3-none-any.whl (53 kB) Collecting Mako Downloading Mako-1.3.5-py3-none-any.whl (78 kB) Collecting typing-extensions>=4 Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) Collecting urllib3>=1.26.0 Downloading urllib3-2.2.3-py3-none-any.whl (126 kB) Collecting itsdangerous>=2.1.2 Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB) Collecting Werkzeug>=3.0.0 Downloading werkzeug-3.0.4-py3-none-any.whl (227 kB) Collecting blinker>=1.6.2 Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB) Collecting graphql-relay<3.3,>=3.1 Downloading graphql_relay-3.2.0-py3-none-any.whl (16 kB) Collecting aniso8601<10,>=8 Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB) Collecting graphql-core<3.3,>=3.1 Downloading graphql_core-3.2.4-py3-none-any.whl (203 kB) Collecting MarkupSafe>=2.0 Downloading MarkupSafe-3.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB) Collecting pyparsing>=2.3.1 Downloading pyparsing-3.1.4-py3-none-any.whl (104 kB) Collecting fonttools>=4.22.0 Downloading fonttools-4.54.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB) Collecting cycler>=0.10 Downloading cycler-0.12.1-py3-none-any.whl (8.3 kB) Collecting kiwisolver>=1.3.1 Downloading kiwisolver-1.4.7-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB) Collecting pillow>=8 Downloading pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB) Collecting contourpy>=1.0.1 Downloading contourpy-1.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (322 kB) Collecting six>=1.5 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting greenlet!=0.4.17 Downloading greenlet-3.1.1-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (599 kB) Collecting jaraco.functools Downloading jaraco.functools-4.1.0-py3-none-any.whl (10 kB) Collecting more-itertools Downloading more_itertools-10.5.0-py3-none-any.whl (60 kB) Collecting autocommand Downloading autocommand-2.2.2-py3-none-any.whl (19 kB) Collecting jaraco.context>=4.1 Downloading jaraco.context-6.0.1-py3-none-any.whl (6.8 kB) Collecting google-auth~=2.0 Downloading google_auth-2.35.0-py2.py3-none-any.whl (208 kB) Collecting gitdb<5,>=4.0.1 Downloading gitdb-4.0.11-py3-none-any.whl (62 kB) Collecting zipp>=3.20 Downloading zipp-3.20.2-py3-none-any.whl (9.2 kB) Collecting backports.tarfile Downloading backports.tarfile-1.2.0-py3-none-any.whl (30 kB) Collecting importlib-metadata!=4.7.0,<9,>=3.7.0 Downloading importlib_metadata-8.4.0-py3-none-any.whl (26 kB) Collecting deprecated>=1.2.6 Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB) Collecting opentelemetry-semantic-conventions==0.48b0 Downloading opentelemetry_semantic_conventions-0.48b0-py3-none-any.whl (149 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB) Collecting certifi>=2017.4.17 Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) Collecting idna<4,>=2.5 Downloading idna-3.10-py3-none-any.whl (70 kB) Collecting wrapt<2,>=1.10 Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB) Collecting smmap<6,>=3.0.1 Downloading smmap-5.0.1-py3-none-any.whl (24 kB) Collecting rsa<5,>=3.1.4 Downloading rsa-4.9-py3-none-any.whl (34 kB) Collecting pyasn1-modules>=0.2.1 Downloading pyasn1_modules-0.4.1-py3-none-any.whl (181 kB) Collecting pyasn1<0.7.0,>=0.4.6 Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB) Building wheels for collected packages: psutil Building wheel for psutil (setup.py): started Building wheel for psutil (setup.py): finished with status 'done' Created wheel for psutil: filename=psutil-5.8.0-cp310-cp310-linux_x86_64.whl size=229611 sha256=72ee154aa9763cd0ed84a9a5f203ae36bf1a6e13ecb7d149cdc2a8f3c86f8a37 Stored in directory: /home/dockeruser/.cache/pip/wheels/12/a3/6d/615295409067d58a62a069d30d296d61d3ac132605e3a9555c Successfully built psutil Installing collected packages: zipp, wrapt, pyasn1, importlib-metadata, deprecated, urllib3, smmap, rsa, pyasn1-modules, opentelemetry-api, idna, charset-normalizer, certifi, cachetools, typing-extensions, six, requests, opentelemetry-semantic-conventions, numpy, more-itertools, MarkupSafe, greenlet, graphql-core, google-auth, gitdb, backports.tarfile, Werkzeug, threadpoolctl, sqlparse, sqlalchemy, scipy, pyyaml, pytz, python-dateutil, pyparsing, protobuf, pillow, packaging, opentelemetry-sdk, Mako, kiwisolver, joblib, Jinja2, jaraco.functools, jaraco.context, itsdangerous, graphql-relay, gitpython, fonttools, databricks-sdk, cycler, contourpy, cloudpickle, click, blinker, autocommand, aniso8601, scikit-learn, pyarrow, pandas, mlflow-skinny, matplotlib, markdown, jaraco.text, gunicorn, graphene, Flask, docker, backports.weakref, alembic, psutil, mlflow, jaraco-collections, backports-tempfile Successfully installed Flask-3.0.3 Jinja2-3.1.4 Mako-1.3.5 MarkupSafe-3.0.1 Werkzeug-3.0.4 alembic-1.13.3 aniso8601-9.0.1 autocommand-2.2.2 backports-tempfile-1.0 backports.tarfile-1.2.0 backports.weakref-1.0.post1 blinker-1.8.2 cachetools-5.5.0 certifi-2024.8.30 charset-normalizer-3.4.0 click-8.1.7 cloudpickle-2.2.1 contourpy-1.3.0 cycler-0.12.1 databricks-sdk-0.34.0 deprecated-1.2.14 docker-7.1.0 fonttools-4.54.1 gitdb-4.0.11 gitpython-3.1.43 google-auth-2.35.0 graphene-3.3 graphql-core-3.2.4 graphql-relay-3.2.0 greenlet-3.1.1 gunicorn-23.0.0 idna-3.10 importlib-metadata-8.4.0 itsdangerous-2.2.0 jaraco-collections-5.1.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jaraco.text-4.0.0 joblib-1.4.2 kiwisolver-1.4.7 markdown-3.7 matplotlib-3.9.2 mlflow-2.16.2 mlflow-skinny-2.16.2 more-itertools-10.5.0 numpy-2.1.2 opentelemetry-api-1.27.0 opentelemetry-sdk-1.27.0 opentelemetry-semantic-conventions-0.48b0 packaging-24.1 pandas-1.5.3 pillow-10.4.0 protobuf-5.28.2 psutil-5.8.0 pyarrow-17.0.0 pyasn1-0.6.1 pyasn1-modules-0.4.1 pyparsing-3.1.4 python-dateutil-2.9.0.post0 pytz-2024.2 pyyaml-6.0.2 requests-2.32.3 rsa-4.9 scikit-learn-1.5.2 scipy-1.14.1 six-1.16.0 smmap-5.0.1 sqlalchemy-2.0.35 sqlparse-0.5.1 threadpoolctl-3.5.0 typing-extensions-4.12.2 urllib3-2.2.3 wrapt-1.16.0 zipp-3.20.2

done #

To activate this environment, use

#

$ conda activate userenv

#

To deactivate an active environment, use

#

$ conda deactivate

WARNING: Package(s) not found: azureml-inference-server-http 2024-10-10T11:05:40,620470267+00:00 | gunicorn/run | Install azureml-inference-server-http version 1.3.0 Collecting azureml-inference-server-http==1.3.0 Downloading azureml_inference_server_http-1.3.0-py3-none-any.whl (42 kB) Collecting inference-schema~=1.8.0 Downloading inference_schema-1.8-py3-none-any.whl (21 kB) Collecting flask-cors~=3.0.1 Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB) Collecting flask<=2.3.2 Downloading Flask-2.3.2-py3-none-any.whl (96 kB) Collecting opencensus-ext-azure~=1.1.0 Downloading opencensus_ext_azure-1.1.13-py2.py3-none-any.whl (43 kB) Requirement already satisfied: werkzeug>=3.0.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from azureml-inference-server-http==1.3.0) (3.0.4) Collecting pydantic-settings Downloading pydantic_settings-2.5.2-py3-none-any.whl (26 kB) Collecting gunicorn==22.0.0 Downloading gunicorn-22.0.0-py3-none-any.whl (84 kB) Collecting pydantic~=2.7.1 Downloading pydantic-2.7.4-py3-none-any.whl (409 kB) Requirement already satisfied: packaging in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from gunicorn==22.0.0->azureml-inference-server-http==1.3.0) (24.1) Requirement already satisfied: click>=8.1.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (8.1.7) Requirement already satisfied: itsdangerous>=2.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (2.2.0) Requirement already satisfied: blinker>=1.6.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (1.8.2) Requirement already satisfied: Jinja2>=3.1.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask<=2.3.2->azureml-inference-server-http==1.3.0) (3.1.4) Requirement already satisfied: Six in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from flask-cors~=3.0.1->azureml-inference-server-http==1.3.0) (1.16.0) Requirement already satisfied: python-dateutil>=2.5.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2.9.0.post0) Requirement already satisfied: wrapt<=1.16.0,>=1.14.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (1.16.0) Requirement already satisfied: pytz>=2017.2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from inference-schema~=1.8.0->azureml-inference-server-http==1.3.0) (2024.2) Collecting azure-core<2.0.0,>=1.12.0 Downloading azure_core-1.31.0-py3-none-any.whl (197 kB) Requirement already satisfied: requests>=2.19.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.32.3) Collecting azure-identity<2.0.0,>=1.5.0 Downloading azure_identity-1.19.0-py3-none-any.whl (187 kB) Collecting opencensus<1.0.0,>=0.11.4 Downloading opencensus-0.11.4-py2.py3-none-any.whl (128 kB) Requirement already satisfied: psutil>=5.6.3 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.8.0) Collecting pydantic-core==2.18.4 Downloading pydantic_core-2.18.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB) Collecting annotated-types>=0.4.0 Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB) Requirement already satisfied: typing-extensions>=4.6.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pydantic~=2.7.1->azureml-inference-server-http==1.3.0) (4.12.2) Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from werkzeug>=3.0.3->azureml-inference-server-http==1.3.0) (3.0.1) Collecting python-dotenv>=0.21.0 Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB) Collecting msal-extensions>=1.2.0 Downloading msal_extensions-1.2.0-py3-none-any.whl (19 kB) Collecting cryptography>=2.5 Downloading cryptography-43.0.1-cp39-abi3-manylinux_2_28_x86_64.whl (4.0 MB) Collecting msal>=1.30.0 Downloading msal-1.31.0-py3-none-any.whl (113 kB) Collecting opencensus-context>=0.1.3 Downloading opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB) Collecting google-api-core<3.0.0,>=1.0.0 Downloading google_api_core-2.21.0-py3-none-any.whl (156 kB) Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.2.3) Requirement already satisfied: idna<4,>=2.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.10) Requirement already satisfied: certifi>=2017.4.17 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2024.8.30) Requirement already satisfied: charset-normalizer<4,>=2 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from requests>=2.19.0->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (3.4.0) Collecting cffi>=1.12 Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB) Collecting proto-plus<2.0.0dev,>=1.22.3 Downloading proto_plus-1.24.0-py3-none-any.whl (50 kB) Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.19.5 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.28.2) Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (2.35.0) Collecting googleapis-common-protos<2.0.dev0,>=1.56.2 Downloading googleapis_common_protos-1.65.0-py2.py3-none-any.whl (220 kB) Collecting PyJWT[crypto]<3,>=1.0.0 Downloading PyJWT-2.9.0-py3-none-any.whl (22 kB) Collecting portalocker<3,>=1.4 Downloading portalocker-2.10.1-py3-none-any.whl (18 kB) Collecting pycparser Downloading pycparser-2.22-py3-none-any.whl (117 kB) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (5.5.0) Requirement already satisfied: rsa<5,>=3.1.4 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (4.9) Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.4.1) Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /opt/miniconda/envs/userenv/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus<1.0.0,>=0.11.4->opencensus-ext-azure~=1.1.0->azureml-inference-server-http==1.3.0) (0.6.1) Installing collected packages: pycparser, cffi, PyJWT, cryptography, proto-plus, portalocker, msal, googleapis-common-protos, pydantic-core, opencensus-context, msal-extensions, google-api-core, azure-core, annotated-types, python-dotenv, pydantic, opencensus, flask, azure-identity, pydantic-settings, opencensus-ext-azure, inference-schema, gunicorn, flask-cors, azureml-inference-server-http Attempting uninstall: flask Found existing installation: flask 3.0.3 Uninstalling flask-3.0.3: Successfully uninstalled flask-3.0.3 Attempting uninstall: gunicorn Found existing installation: gunicorn 23.0.0 Uninstalling gunicorn-23.0.0: Successfully uninstalled gunicorn-23.0.0 Successfully installed PyJWT-2.9.0 annotated-types-0.7.0 azure-core-1.31.0 azure-identity-1.19.0 azureml-inference-server-http-1.3.0 cffi-1.17.1 cryptography-43.0.1 flask-2.3.2 flask-cors-3.0.10 google-api-core-2.21.0 googleapis-common-protos-1.65.0 gunicorn-22.0.0 inference-schema-1.8 msal-1.31.0 msal-extensions-1.2.0 opencensus-0.11.4 opencensus-context-0.1.3 opencensus-ext-azure-1.1.13 portalocker-2.10.1 proto-plus-1.24.0 pycparser-2.22 pydantic-2.7.4 pydantic-core-2.18.4 pydantic-settings-2.5.2 python-dotenv-1.0.1 2024-10-10T11:06:18,005275637+00:00 | gunicorn/run | 2024-10-10T11:06:18,008905681+00:00 | gunicorn/run | ############################################### 2024-10-10T11:06:18,012396323+00:00 | gunicorn/run | Checking if the Python package azureml-inference-server-http is installed 2024-10-10T11:06:18,015871665+00:00 | gunicorn/run | ############################################### 2024-10-10T11:06:18,019106304+00:00 | gunicorn/run | 2024-10-10T11:06:22,619276944+00:00 | gunicorn/run | 2024-10-10T11:06:22,622712786+00:00 | gunicorn/run | ############################################### 2024-10-10T11:06:22,625700722+00:00 | gunicorn/run | AzureML Inference Server 2024-10-10T11:06:22,628489655+00:00 | gunicorn/run | ############################################### 2024-10-10T11:06:22,631337490+00:00 | gunicorn/run | 2024-10-10T11:06:22,634345926+00:00 | gunicorn/run | Starting AzureML Inference Server HTTP. 2024-10-10 11:06:23,311 I [12] azmlinfsrv - Loaded logging config from /opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/logging.json 2024-10-10 11:06:23,408 I [12] gunicorn.error - Starting gunicorn 22.0.0 2024-10-10 11:06:23,410 I [12] gunicorn.error - Listening at: http://0.0.0.0:31311 (12) 2024-10-10 11:06:23,410 I [12] gunicorn.error - Using worker: sync 2024-10-10 11:06:23,414 I [152] gunicorn.error - Booting worker with pid: 152 /opt/miniconda/envs/userenv/lib/python3.10/site-packages/pydantic/_internal/_fields.py:160: UserWarning: Field "model_dc_storageenabled" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = ('settings_',). warnings.warn(

Azure ML Inferencing HTTP server v1.3.0

Server Settings

Entry Script Name: /var/mlflow_resources/mlflow_score_script.py Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1 Config File: None Worker Count: 1 Worker Timeout (seconds): 300 Server Port: 31311 Health Port: 31311 Application Insights Enabled: false Application Insights Key: None Inferencing HTTP server version: azmlinfsrv/1.3.0 CORS for the specified origins: None Create dedicated endpoint for health: None

Server Routes

Liveness Probe: GET 127.0.0.1:31311/ Score: POST 127.0.0.1:31311/score

Warnings

Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10 11:06:23,996 W [152] azmlinfsrv - Found extra keys in the config file that are not supported by the server. Extra keys = ['AZUREML_ENTRY_SCRIPT', 'AZUREML_MODEL_DIR', 'HOSTNAME'] 2024-10-10 11:06:25,124 I [152] azmlinfsrv - AML_FLASK_ONE_COMPATIBILITY is set. Patched Flask to ensure compatibility with Flask 1. Initializing logger 2024-10-10 11:06:25,127 I [152] azmlinfsrv - Starting up app insights client 2024-10-10 11:06:26,402 E [152] azmlinfsrv - Traceback (most recent call last): File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 77, in load_script main_module_spec.loader.exec_module(user_module) File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/var/mlflow_resources/mlflow_score_script.py", line 5, in import mlflow File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/init.py", line 34, in from mlflow import ( File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/init.py", line 5, in from mlflow.data import dataset_registry File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/dataset_registry.py", line 137, in from mlflow.data.pandas_dataset import from_pandas File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/mlflow/data/pandas_dataset.py", line 6, in import pandas as pd File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/init.py", line 22, in from pandas.compat import is_numpy_dev as _is_numpy_dev # pyright: ignore # noqa:F401 File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/init.py", line 18, in from pandas.compat.numpy import ( File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/compat/numpy/init.py", line 4, in from pandas.util.version import Version File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/init.py", line 2, in from pandas.util._decorators import ( # noqa:F401 File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in from pandas._libs.properties import cache_readonly File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/pandas/_libs/init.py", line 13, in from pandas._libs.interval import Interval File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 91, in setup self.user_script.load_script(config.app_root) File "/opt/miniconda/envs/userenv/lib/python3.10/site-packages/azureml_inference_server_http/server/user_script.py", line 79, in load_script raise UserScriptImportException(ex) from ex azureml_inference_server_http.server.user_script.UserScriptImportException: Failed to import user script because it raised an unhandled exception

2024-10-10 11:06:26,404 I [152] gunicorn.error - Worker exiting (pid: 152) 2024-10-10 11:06:27,130 E [12] gunicorn.error - Worker (pid:152) exited with code 3 2024-10-10 11:06:27,132 E [12] gunicorn.error - Shutting down: Master 2024-10-10 11:06:27,133 E [12] gunicorn.error - Reason: Worker failed to boot.

Azure ML Inferencing HTTP server v1.3.0

Server Settings

Entry Script Name: /var/mlflow_resources/mlflow_score_script.py Model Directory: /var/azureml-app/azureml-models/credit_defaults_model/1 Config File: None Worker Count: 1 Worker Timeout (seconds): 300 Server Port: 31311 Health Port: 31311 Application Insights Enabled: false Application Insights Key: None Inferencing HTTP server version: azmlinfsrv/1.3.0 CORS for the specified origins: None Create dedicated endpoint for health: None

Server Routes

Liveness Probe: GET 127.0.0.1:31311/ Score: POST 127.0.0.1:31311/score

Warnings

Azmlinfsrv will be migrating to Pydantic 2.0 on 1/15/24. This is a breaking change for any Pydantic 1.0 code.

2024-10-10T11:06:27,400529401+00:00 - gunicorn/finish 3 0 2024-10-10T11:06:27,404048457+00:00 - Exit code 3 is not normal. Killing image.



### Expected behavior

The endpoint is provisioned successfully. 

### Actual behavior

Provisioning State = Failed

### Addition information

According to [this](https://github.com/pandas-dev/pandas/issues/59270) reference, the underlying error condition showed in the logs (**ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject**) is due to an incompatibility between the _numpy_ and _pandas_ package versions. Could this discrepancy be coming from the environment image (i.e. `environment="azureml://registries/azureml/environments/sklearn-1.5/labels/latest"`)? Or is it somehow related to my local environment?
riccardotrevisan commented 1 week ago

As a workaround, I created a custom environment without specifying package versions:

name: sklearn-custom
channels:
  - conda-forge
  - anaconda
dependencies:
  - python=3.10
  - pip
  - pandas
  - scipy
  - numpy
  - pip:
      - scikit-learn-intelex
      - azureml-core
      - azureml-defaults
      - azureml-mlflow
      - azureml-telemetry
      - scikit-learn
      - joblib
      - py-spy
      - debugpy
      - ipykernel
      - tensorboard
      - psutil
      - matplotlib
      - tqdm
      - py-cpuinfo
      - torch-tb-profiler

Using it as training environment in the Getting Started notebook apparently allows to deploy the model on the endpoint correctly.

Neverthless, Azure dev team should look into this and fix the tutorial, at least.