hey team, im getting deployment issues, i have used docker file-based deployment as your sample document suggests i used same training process as your sample script suggest. I got the same error in ACI as Local deployment.

code

import os from feast import FeatureStore from azureml.core import Workspace

access key vault to get secrets

ws = Workspace.from_config() kv = ws.get_default_keyvault()

os.environ['REGISTRY_PATH']=kv.get_secret("FEAST-REGISTRY-PATH") os.environ['SQL_CONN']=kv.get_secret("FEAST-OFFLINE-STORE-CONN") os.environ['REDIS_CONN']=kv.get_secret("FEAST-ONLINE-STORE-CONN") fs = FeatureStore("./feature_repo")

from azureml.core.environment import Environment from azureml.core.webservice import AciWebservice from azureml.core import Workspace

ws = Workspace.from_config() keyvault = ws.get_default_keyvault()

get registered environment

env = Environment("feast-env") env.docker.base_image = None env.docker.base_dockerfile = "./inference.dockerfile" env.python.user_managed_dependencies = True env.inferencing_stack_version = 'latest'

env.python.interpreter_path ="/anaconda/envs/azureml_py38/bin/python"#"/azureml-envs/feast/bin/python"

again ensure that the scoring environment has access to the registry file

env.environment_variables = { "SQL_CONN": fs.config.offline_store.connection_string, "REDIS_CONN": fs.config.online_store.connection_string, "REGISTRY_PATH": fs.config.registry.path, "AZURE_CLIENT_ID":"client_id", "AZURE_TENANT_ID":"tenant_id", "AZURE_CLIENT_SECRET":"client_secret" }

import uuid from azureml.core.model import InferenceConfig from azureml.core.environment import Environment from azureml.core.model import Model

get the registered model

model = Model(ws, "az_order_model")

create an inference config i.e. the scoring script and environment

inference_config = InferenceConfig( entry_script="score.py", environment=env, source_directory="src" )

Local Deployment

from azureml.core.webservice import LocalWebservice

This is optional, if not provided Docker will choose a random unused port.

deployment_config = LocalWebservice.deploy_configuration(port=6789)

local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config)

local_service.wait_for_deployment()

Docker File:

FROM mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210615.v1

ENV AZUREML_CONDA_ENVIRONMENT_PATH /azureml-envs/feast

RUN conda create -p $AZUREML_CONDA_ENVIRONMENT_PATH \ python=3.8 pip=20.2.4

ENV PATH $AZUREML_CONDA_ENVIRONMENT_PATH/bin:$PATH

RUN apt-get install -y gcc RUN apt-get install -y unixodbc-dev

RUN pip install \ 'protobuf>=3.10,<3.20' \ 'azureml-defaults==1.42.0' \ 'feast-azure-provider==0.3.0' \ 'scikit-learn==0.22.2.post1' \ 'joblib===1.1.0' \ 'itsdangerous==2.0.1'

ERROR

/tmp/ipykernel_7361/3392805753.py:6: FutureWarning: azureml.core.model: To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint, please refer to respective documentations https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints / https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere For more information on migration, see https://aka.ms/acimoemigration. To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False' local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config) 02/13/2023 12:49:33 PM ERROR:Error: Container has crashed. Did your init method fail?

Downloading model az_order_model:2 to /tmp/azureml_dhenq806/az_order_model/2 Generating Docker build context. Package creation Succeeded Logging into Docker registry acoreus2mlopsdev01.azurecr.io Logging into Docker registry acoreus2mlopsdev01.azurecr.io Building Docker image from Dockerfile... Step 1/5 : FROM acoreus2mlopsdev01.azurecr.io/azureml/azureml_95f2f8f3e14df742ae3060df5fc359e1 ---> 0bb6bba950ba Step 2/5 : COPY azureml-app /var/azureml-app ---> f81a926f4fbe Step 3/5 : RUN mkdir -p '/var/azureml-app' && echo eyJhY2NvdW50Q29udGV4dCI6eyJzdWJzY3JpcHRpb25JZCI6ImViNWEyNTdhLTc2ZTItNDcyZC05ZmRjLTdmOWIzMTcyZmNmNiIsInJlc291cmNlR3JvdXBOYW1lIjoia2Fnb29sLW1sb3BzLWRldi1yZyIsImFjY291bnROYW1lIjoibWx3cC1tbG9wcy1ldXMyLWRldi0wMSIsIndvcmtzcGFjZUlkIjoiZjQ4OGFhM2YtYWIyNC00NGMzLTkyYWYtMjVlMDYyZDRmODU1In0sIm1vZGVscyI6e30sIm1vZGVsc0luZm8iOnt9fQ== | base64 --decode > /var/azureml-app/model_config_map.json ---> Running in f946e4625c9f ---> 01293bca6b6d Step 4/5 : RUN mv '/var/azureml-app/tmps1p_y_b4.py' /var/azureml-app/main.py ---> Running in 99d2feb73dd0 ---> 2c4cc4bb0b88 Step 5/5 : CMD ["runsvdir","/var/runit"] ---> Running in ee9868b848e3 ---> dd11da4609e3 Successfully built dd11da4609e3 Successfully tagged test:latest Starting Docker container... Docker container running. Checking container health...

Container Logs: 2023-02-13T12:48:59,136988351+00:00 - iot-server/run 2023-02-13T12:48:59,139444185+00:00 - nginx/run 2023-02-13T12:48:59,141341611+00:00 - rsyslog/run 2023-02-13T12:48:59,153172974+00:00 - gunicorn/run 2023-02-13T12:48:59,156998827+00:00 | gunicorn/run | 2023-02-13T12:48:59,158928354+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,160549477+00:00 | gunicorn/run | AzureML Container Runtime Information 2023-02-13T12:48:59,162503104+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,164729834+00:00 | gunicorn/run | 2023-02-13T12:48:59,166495459+00:00 | gunicorn/run | 2023-02-13T12:48:59,168390485+00:00 | gunicorn/run | PATH environment variable: /azureml-envs/feast/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-02-13T12:48:59,170397713+00:00 | gunicorn/run | PYTHONPATH environment variable: 2023-02-13T12:48:59,171754532+00:00 | gunicorn/run | 2023-02-13T12:48:59,173140651+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)

EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting... 2023-02-13T12:48:59,215335234+00:00 - iot-server/finish 1 0 2023-02-13T12:48:59,216607852+00:00 - Exit code 1 is normal. Not restarting iot-server. absl-py==1.4.0 adal==1.2.7 anyio==3.6.2 applicationinsights==0.11.10 argcomplete==2.0.0 async-timeout==4.0.2 attrs==22.2.0 azure-common==1.1.28 azure-core==1.22.1 azure-graphrbac==0.61.1 azure-identity==1.12.0 azure-mgmt-authorization==2.0.0 azure-mgmt-containerregistry==9.1.0 azure-mgmt-core==1.3.2 azure-mgmt-keyvault==9.3.0 azure-mgmt-resource==21.0.0 azure-mgmt-storage==20.0.0 azure-storage-blob==12.14.1 azureml-core==1.42.0.post1 azureml-dataprep==4.0.4 azureml-dataprep-native==38.0.0 azureml-dataprep-rslex==2.6.3 azureml-dataset-runtime==1.42.0 azureml-defaults==1.42.0 azureml-inference-server-http==0.4.14 backports.tempfile==1.0 backports.weakref==1.0.post1 bcrypt==4.0.1 cachetools==5.3.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.15.1 charset-normalizer==3.0.1 click==7.1.2 cloudpickle==2.2.1 colorama==0.4.6 configparser==3.7.4 contextlib2==21.6.0 cryptography==36.0.2 dill==0.3.4 distro==1.8.0 docker==5.0.3 dotnetcore2==3.1.23 fastapi==0.89.1 fastavro==1.7.1 feast==0.18.1 feast-azure-provider==0.3.0 Flask==1.0.3 fusepy==3.0.1 google-api-core==2.11.0 google-auth==2.16.0 googleapis-common-protos==1.52.0 greenlet==2.0.2 grpcio==1.51.1 grpcio-reflection==1.51.1 gunicorn==20.1.0 h11==0.14.0 hiredis==2.2.1 httptools==0.5.0 humanfriendly==10.0 idna==3.4 importlib-resources==5.10.2 inference-schema==1.3.0 isodate==0.6.1 itsdangerous==2.0.1 jeepney==0.8.0 Jinja2==3.0.3 jmespath==1.0.0 joblib==1.1.0 json-logging-py==0.2 jsonpickle==2.2.0 jsonschema==4.17.3 knack==0.9.0 MarkupSafe==2.1.2 mmh3==3.0.0 msal==1.21.0 msal-extensions==1.0.0 msrest==0.6.21 msrestazure==0.6.4 ndg-httpsclient==0.5.1 numpy==1.24.2 oauthlib==3.2.2 opencensus==0.11.1 opencensus-context==0.1.3 opencensus-ext-azure==1.1.8 packaging==21.3 pandas==1.5.3 pandavro==1.5.2 paramiko==2.12.0 pathspec==0.11.0 pkginfo==1.9.6 pkgutil-resolve-name==1.3.10 portalocker==2.7.0 proto-plus==1.19.6 protobuf==3.19.6 psutil==5.9.4 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 pydantic==1.10.4 Pygments==2.14.0 PyJWT==2.6.0 PyNaCl==1.5.0 pyodbc==4.0.35 pyOpenSSL==22.1.0 pyparsing==3.0.9 pyrsistent==0.19.3 PySocks==1.7.1 python-dateutil==2.8.2 python-dotenv==0.21.1 pytz==2022.7.1 PyYAML==6.0 redis==4.4.2 requests==2.28.2 requests-oauthlib==1.3.1 rsa==4.9 scikit-learn==0.22.2.post1 scipy==1.10.0 SecretStorage==3.3.3 six==1.16.0 sniffio==1.3.0 SQLAlchemy==2.0.1 starlette==0.22.0 tabulate==0.8.10 tenacity==8.1.0 tensorflow-metadata==1.12.0 toml==0.10.2 tqdm==4.64.1 typing-extensions==4.4.0 urllib3==1.26.9 uvicorn==0.20.0 uvloop==0.17.0 watchfiles==0.18.1 websocket-client==1.5.1 websockets==10.4 Werkzeug==1.0.1 wrapt==1.12.1 zipp==3.12.1

Azure ML Inferencing HTTP server v0.4.14

Server Settings

Entry Script Name: score.py Model Directory: azureml-models/az_order_model/2 Worker Count: 1 Worker Timeout (seconds): 300 Server Port: 31311 Application Insights Enabled: false Application Insights Key: None

Server Routes

Liveness Probe: GET 127.0.0.1:31311/ Score: POST 127.0.0.1:31311/score

Starting gunicorn 20.1.0 Listening at: http://0.0.0.0:31311/ (13) Using worker: sync Booting worker with pid: 64 Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.scriptrun = azureml.core.script_run:ScriptRun._from_run_dto with exception (cryptography 36.0.2 (/azureml-envs/feast/lib/python3.8/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'pyopenssl'}). Initializing logger 2023-02-13 12:49:02,162 | root | INFO | Starting up app insights client logging socket was found. logging is available. logging socket was found. logging is available. 2023-02-13 12:49:02,162 | root | INFO | Starting up request id generator 2023-02-13 12:49:02,163 | root | INFO | Starting up app insight hooks 2023-02-13 12:49:02,163 | root | INFO | Invoking user's init function connecting to registry... no request id,connecting to registry...

connecting to repo config... no request id,connecting to repo config...

connecting to feature store... no request id,connecting to feature store...

2023-02-13 12:49:02,174 | root | ERROR | User's init function failed 2023-02-13 12:49:02,179 | root | ERROR | Encountered Exception Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 21, in init from azure.storage.blob import BlobClient File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/init.py", line 10, in from ._blob_client import BlobClient File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 24, in from ._shared.base_client import StorageAccountHostsMixin, parse_connection_str, parse_query, TransportWrapper File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 40, in from .constants import CONNECTION_TIMEOUT, READ_TIMEOUT, SERVICE_HOST_BASE File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/constants.py", line 7, in from .._serialize import _SUPPORTED_API_VERSIONS File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_serialize.py", line 18, in from ._models import ( File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_models.py", line 18, in from ._generated.models import ArrowField File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/init.py", line 9, in from ._azure_blob_storage import AzureBlobStorage File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_azure_blob_storage.py", line 15, in from . import models File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/init.py", line 9, in from ._models_py3 import AccessPolicy File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/_models_py3.py", line 14, in from .. import _serialization File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_serialization.py", line 51, in from azure.core.exceptions import DeserializationError, SerializationError, raise_with_traceback ImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 201, in register main.init() File "/var/azureml-app/src/score.py", line 126, in init store = FeatureStore(config=repo_cfg) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 338, in wrapper raise exc.with_traceback(traceback) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 324, in wrapper return func(*args, **kwargs) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/feature_store.py", line 127, in init self._registry = Registry(registry_config, repo_path=self.repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/registry.py", line 166, in init self._registry_store = cls(registry_config, repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 28, in init raise FeastExtrasDependencyImportError("az", str(e)) feast.errors.FeastExtrasDependencyImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py) You may need run pip install 'feast[az]'

2023-02-13 12:49:02,179 | root | INFO | Waiting for logs to be sent to Application Insights before exit. 2023-02-13 12:49:02,179 | root | INFO | Waiting 30 seconds for upload. /azureml-envs/feast/lib/python3.8/site-packages/azureml/data/dataset_type_definitions.py:67: DeprecationWarning: FileType Enum is Deprecated in > 1.0.39. Use strings instead. warnings.warn( Worker exiting (pid: 64) Shutting down: Master Reason: Worker failed to boot. 2023-02-13T12:49:32,454322839+00:00 - gunicorn/finish 3 0 2023-02-13T12:49:32,455683858+00:00 - Exit code 3 is not normal. Killing image.

WebserviceException Traceback (most recent call last) Cell In[4], line 8 4 deployment_config = LocalWebservice.deploy_configuration(port=6789) 6 local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config) ----> 8 local_service.wait_for_deployment()

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/local.py:72, in _in_state..decorator..decorated(self, *args, *kwargs) 69 if self.state not in states: 70 raise WebserviceException('Cannot call {}() when service is {}.'.format(func.name, self.state), 71 logger=module_logger) ---> 72 return func(self, args, **kwargs)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/local.py:611, in LocalWebservice.wait_for_deployment(self, show_output) 603 """ 604 Poll the running LocalWebservice deployment. 605 (...) 608 :raises: :class:azureml.exceptions.WebserviceException 609 """ 610 try: --> 611 container_health_check(self._port, 612 self._container, 613 health_url=self._get_health_url(), 614 cleanup_if_failed=False) 616 self.state = LocalWebservice.STATE_RUNNING 618 print('Local webservice is {} at {}'.format(self.state, self._base_url))

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:748, in container_health_check(docker_port, container, health_url, cleanup_if_failed) 745 continue 746 elif container.status == 'exited': 747 # The container has started and crashed. --> 748 _raise_for_container_failure(container, cleanup_if_failed, 749 'Error: Container has crashed. Did your init method fail?') 751 # The container hasn't crashed, so try to ping the health endpoint. 752 try:

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:1265, in _raise_for_container_failure(container, cleanup, message) 1262 if cleanup: 1263 cleanup_container(container) -> 1265 raise WebserviceException(message, logger=module_logger)

WebserviceException: WebserviceException: Message: Error: Container has crashed. Did your init method fail? InnerException None ErrorResponse { "error": { "message": "Error: Container has crashed. Did your init method fail?" } }

SCORE SCRIPT

import os import logging import json import numpy import joblib import pandas as pd from datetime import timedelta

import os import logging import json import joblib from feast import FeatureStore, RepoConfig from feast.registry import RegistryConfig from feast_azure_provider.mssqlserver import MsSqlServerOfflineStoreConfig from feast.infra.online_stores.redis import RedisOnlineStoreConfig, RedisOnlineStore from azureml.core.model import Model from azureml.core import Workspace, Dataset, Datastore

def init(): sql_conn_str = os.getenv("SQL_CONN") redis_conn_str = os.getenv("REDIS_CONN") feast_registry_path = os.getenv("REGISTRY_PATH")

print("connecting to registry...")
reg_config = RegistryConfig(
    registry_store_type="feast_azure_provider.registry_store.AzBlobRegistryStore",
    path=feast_registry_path,
)

print("connecting to repo config...")
repo_cfg = RepoConfig(
    project="production",
    provider="feast_azure_provider.azure_provider.AzureProvider",
    registry=reg_config,
    offline_store=MsSqlServerOfflineStoreConfig(connection_string=sql_conn_str),
    online_store=RedisOnlineStoreConfig(connection_string=redis_conn_str),
)
global store
print("connecting to feature store...")
store = FeatureStore(config=repo_cfg)

global model
# AZUREML_MODEL_DIR is an environment variable created during deployment.
# It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)
model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model/model_az38.pkl")
# deserialize the model file back into a sklearn model
model = joblib.load(model_path)
print("read model, init complete")

def run(raw_data): data = json.loads(raw_data) feature_vector = store.get_online_features( features=[ "driver_stats:conv_rate", "driver_stats:avg_daily_trips", "driver_stats:acc_rate", "customer_profile:current_balance", "customer_profile:avg_passenger_count", "customer_profile:lifetime_trip_count", ], entity_rows=[data], ).to_df() logging.info(feature_vector) if len(feature_vector.dropna()) > 0: data = feature_vector[ [ "conv_rate", "avg_daily_trips", "acc_rate", "current_balance", "avg_passenger_count", "lifetime_trip_count", ] ]

    y_hat = model.predict(data)
    return y_hat.tolist()
else:
    return 0.0

Note

I have tried different types of deployment, ACI, LOCAL, Managed Online Endpoints in azure but still im facing same problem. i have changed dependencies in docker file same error i have encounter can please suggest where exactly we are missing thanks.

Azure / feast-azure

Deployment issue in AZURE #76