hey team, im getting deployment issues, i have used docker file-based deployment as your sample document suggests i used same training process as your sample script suggest. I got the same error in ACI as Local deployment.
code
import os
from feast import FeatureStore
from azureml.core import Workspace
Liveness Probe: GET 127.0.0.1:31311/
Score: POST 127.0.0.1:31311/score
Starting gunicorn 20.1.0
Listening at: http://0.0.0.0:31311/ (13)
Using worker: sync
Booting worker with pid: 64
Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage
Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.scriptrun = azureml.core.script_run:ScriptRun._from_run_dto with exception (cryptography 36.0.2 (/azureml-envs/feast/lib/python3.8/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'pyopenssl'}).
Initializing logger
2023-02-13 12:49:02,162 | root | INFO | Starting up app insights client
logging socket was found. logging is available.
logging socket was found. logging is available.
2023-02-13 12:49:02,162 | root | INFO | Starting up request id generator
2023-02-13 12:49:02,163 | root | INFO | Starting up app insight hooks
2023-02-13 12:49:02,163 | root | INFO | Invoking user's init function
connecting to registry...
no request id,connecting to registry...
connecting to repo config...
no request id,connecting to repo config...
connecting to feature store...
no request id,connecting to feature store...
2023-02-13 12:49:02,174 | root | ERROR | User's init function failed
2023-02-13 12:49:02,179 | root | ERROR | Encountered Exception Traceback (most recent call last):
File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 21, in init
from azure.storage.blob import BlobClient
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/init.py", line 10, in
from ._blob_client import BlobClient
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 24, in
from ._shared.base_client import StorageAccountHostsMixin, parse_connection_str, parse_query, TransportWrapper
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 40, in
from .constants import CONNECTION_TIMEOUT, READ_TIMEOUT, SERVICE_HOST_BASE
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/constants.py", line 7, in
from .._serialize import _SUPPORTED_API_VERSIONS
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_serialize.py", line 18, in
from ._models import (
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_models.py", line 18, in
from ._generated.models import ArrowField
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/init.py", line 9, in
from ._azure_blob_storage import AzureBlobStorage
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_azure_blob_storage.py", line 15, in
from . import models
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/init.py", line 9, in
from ._models_py3 import AccessPolicy
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/_models_py3.py", line 14, in
from .. import _serialization
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_serialization.py", line 51, in
from azure.core.exceptions import DeserializationError, SerializationError, raise_with_traceback
ImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/azureml-envs/feast/lib/python3.8/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 201, in register
main.init()
File "/var/azureml-app/src/score.py", line 126, in init
store = FeatureStore(config=repo_cfg)
File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 338, in wrapper
raise exc.with_traceback(traceback)
File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 324, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/feast/lib/python3.8/site-packages/feast/feature_store.py", line 127, in init
self._registry = Registry(registry_config, repo_path=self.repo_path)
File "/azureml-envs/feast/lib/python3.8/site-packages/feast/registry.py", line 166, in init
self._registry_store = cls(registry_config, repo_path)
File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 28, in init
raise FeastExtrasDependencyImportError("az", str(e))
feast.errors.FeastExtrasDependencyImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py)
You may need run pip install 'feast[az]'
2023-02-13 12:49:02,179 | root | INFO | Waiting for logs to be sent to Application Insights before exit.
2023-02-13 12:49:02,179 | root | INFO | Waiting 30 seconds for upload.
/azureml-envs/feast/lib/python3.8/site-packages/azureml/data/dataset_type_definitions.py:67: DeprecationWarning: FileType Enum is Deprecated in > 1.0.39. Use strings instead.
warnings.warn(
Worker exiting (pid: 64)
Shutting down: Master
Reason: Worker failed to boot.
2023-02-13T12:49:32,454322839+00:00 - gunicorn/finish 3 0
2023-02-13T12:49:32,455683858+00:00 - Exit code 3 is not normal. Killing image.
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/local.py:72, in _in_state..decorator..decorated(self, *args, *kwargs)
69 if self.state not in states:
70 raise WebserviceException('Cannot call {}() when service is {}.'.format(func.name, self.state),
71 logger=module_logger)
---> 72 return func(self, args, **kwargs)
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:748, in container_health_check(docker_port, container, health_url, cleanup_if_failed)
745 continue
746 elif container.status == 'exited':
747 # The container has started and crashed.
--> 748 _raise_for_container_failure(container, cleanup_if_failed,
749 'Error: Container has crashed. Did your init method fail?')
751 # The container hasn't crashed, so try to ping the health endpoint.
752 try:
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:1265, in _raise_for_container_failure(container, cleanup, message)
1262 if cleanup:
1263 cleanup_container(container)
-> 1265 raise WebserviceException(message, logger=module_logger)
WebserviceException: WebserviceException:
Message: Error: Container has crashed. Did your init method fail?
InnerException None
ErrorResponse
{
"error": {
"message": "Error: Container has crashed. Did your init method fail?"
}
}
SCORE SCRIPT
import os
import logging
import json
import numpy
import joblib
import pandas as pd
from datetime import timedelta
import os
import logging
import json
import joblib
from feast import FeatureStore, RepoConfig
from feast.registry import RegistryConfig
from feast_azure_provider.mssqlserver import MsSqlServerOfflineStoreConfig
from feast.infra.online_stores.redis import RedisOnlineStoreConfig, RedisOnlineStore
from azureml.core.model import Model
from azureml.core import Workspace, Dataset, Datastore
print("connecting to registry...")
reg_config = RegistryConfig(
registry_store_type="feast_azure_provider.registry_store.AzBlobRegistryStore",
path=feast_registry_path,
)
print("connecting to repo config...")
repo_cfg = RepoConfig(
project="production",
provider="feast_azure_provider.azure_provider.AzureProvider",
registry=reg_config,
offline_store=MsSqlServerOfflineStoreConfig(connection_string=sql_conn_str),
online_store=RedisOnlineStoreConfig(connection_string=redis_conn_str),
)
global store
print("connecting to feature store...")
store = FeatureStore(config=repo_cfg)
global model
# AZUREML_MODEL_DIR is an environment variable created during deployment.
# It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)
model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model/model_az38.pkl")
# deserialize the model file back into a sklearn model
model = joblib.load(model_path)
print("read model, init complete")
I have tried different types of deployment, ACI, LOCAL, Managed Online Endpoints in azure but still im facing same problem.
i have changed dependencies in docker file same error i have encounter can please suggest where exactly we are missing thanks.
hey team, im getting deployment issues, i have used docker file-based deployment as your sample document suggests i used same training process as your sample script suggest. I got the same error in ACI as Local deployment.
code
import os from feast import FeatureStore from azureml.core import Workspace
access key vault to get secrets
ws = Workspace.from_config() kv = ws.get_default_keyvault()
os.environ['REGISTRY_PATH']=kv.get_secret("FEAST-REGISTRY-PATH") os.environ['SQL_CONN']=kv.get_secret("FEAST-OFFLINE-STORE-CONN") os.environ['REDIS_CONN']=kv.get_secret("FEAST-ONLINE-STORE-CONN") fs = FeatureStore("./feature_repo")
from azureml.core.environment import Environment from azureml.core.webservice import AciWebservice from azureml.core import Workspace
ws = Workspace.from_config() keyvault = ws.get_default_keyvault()
get registered environment
env = Environment("feast-env") env.docker.base_image = None env.docker.base_dockerfile = "./inference.dockerfile" env.python.user_managed_dependencies = True env.inferencing_stack_version = 'latest'
env.python.interpreter_path ="/anaconda/envs/azureml_py38/bin/python"#"/azureml-envs/feast/bin/python"
again ensure that the scoring environment has access to the registry file
env.environment_variables = { "SQL_CONN": fs.config.offline_store.connection_string, "REDIS_CONN": fs.config.online_store.connection_string, "REGISTRY_PATH": fs.config.registry.path, "AZURE_CLIENT_ID":"client_id", "AZURE_TENANT_ID":"tenant_id", "AZURE_CLIENT_SECRET":"client_secret" }
import uuid from azureml.core.model import InferenceConfig from azureml.core.environment import Environment from azureml.core.model import Model
get the registered model
model = Model(ws, "az_order_model")
create an inference config i.e. the scoring script and environment
inference_config = InferenceConfig( entry_script="score.py", environment=env, source_directory="src" )
Local Deployment
from azureml.core.webservice import LocalWebservice
This is optional, if not provided Docker will choose a random unused port.
deployment_config = LocalWebservice.deploy_configuration(port=6789)
local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config)
local_service.wait_for_deployment()
Docker File:
FROM mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20210615.v1
ENV AZUREML_CONDA_ENVIRONMENT_PATH /azureml-envs/feast
RUN conda create -p $AZUREML_CONDA_ENVIRONMENT_PATH \ python=3.8 pip=20.2.4
ENV PATH $AZUREML_CONDA_ENVIRONMENT_PATH/bin:$PATH
RUN apt-get install -y gcc RUN apt-get install -y unixodbc-dev
RUN pip install \ 'protobuf>=3.10,<3.20' \ 'azureml-defaults==1.42.0' \ 'feast-azure-provider==0.3.0' \ 'scikit-learn==0.22.2.post1' \ 'joblib===1.1.0' \ 'itsdangerous==2.0.1'
ERROR
/tmp/ipykernel_7361/3392805753.py:6: FutureWarning: azureml.core.model: To leverage new model deployment capabilities, AzureML recommends using CLI/SDK v2 to deploy models as online endpoint, please refer to respective documentations https://docs.microsoft.com/azure/machine-learning/how-to-deploy-managed-online-endpoints / https://docs.microsoft.com/azure/machine-learning/how-to-attach-kubernetes-anywhere For more information on migration, see https://aka.ms/acimoemigration. To disable CLI/SDK v1 deprecation warning set AZUREML_LOG_DEPRECATION_WARNING_ENABLED to 'False' local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config) 02/13/2023 12:49:33 PM ERROR:Error: Container has crashed. Did your init method fail?
Downloading model az_order_model:2 to /tmp/azureml_dhenq806/az_order_model/2 Generating Docker build context. Package creation Succeeded Logging into Docker registry acoreus2mlopsdev01.azurecr.io Logging into Docker registry acoreus2mlopsdev01.azurecr.io Building Docker image from Dockerfile... Step 1/5 : FROM acoreus2mlopsdev01.azurecr.io/azureml/azureml_95f2f8f3e14df742ae3060df5fc359e1 ---> 0bb6bba950ba Step 2/5 : COPY azureml-app /var/azureml-app ---> f81a926f4fbe Step 3/5 : RUN mkdir -p '/var/azureml-app' && echo eyJhY2NvdW50Q29udGV4dCI6eyJzdWJzY3JpcHRpb25JZCI6ImViNWEyNTdhLTc2ZTItNDcyZC05ZmRjLTdmOWIzMTcyZmNmNiIsInJlc291cmNlR3JvdXBOYW1lIjoia2Fnb29sLW1sb3BzLWRldi1yZyIsImFjY291bnROYW1lIjoibWx3cC1tbG9wcy1ldXMyLWRldi0wMSIsIndvcmtzcGFjZUlkIjoiZjQ4OGFhM2YtYWIyNC00NGMzLTkyYWYtMjVlMDYyZDRmODU1In0sIm1vZGVscyI6e30sIm1vZGVsc0luZm8iOnt9fQ== | base64 --decode > /var/azureml-app/model_config_map.json ---> Running in f946e4625c9f ---> 01293bca6b6d Step 4/5 : RUN mv '/var/azureml-app/tmps1p_y_b4.py' /var/azureml-app/main.py ---> Running in 99d2feb73dd0 ---> 2c4cc4bb0b88 Step 5/5 : CMD ["runsvdir","/var/runit"] ---> Running in ee9868b848e3 ---> dd11da4609e3 Successfully built dd11da4609e3 Successfully tagged test:latest Starting Docker container... Docker container running. Checking container health...
Container Logs: 2023-02-13T12:48:59,136988351+00:00 - iot-server/run 2023-02-13T12:48:59,139444185+00:00 - nginx/run 2023-02-13T12:48:59,141341611+00:00 - rsyslog/run 2023-02-13T12:48:59,153172974+00:00 - gunicorn/run 2023-02-13T12:48:59,156998827+00:00 | gunicorn/run | 2023-02-13T12:48:59,158928354+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,160549477+00:00 | gunicorn/run | AzureML Container Runtime Information 2023-02-13T12:48:59,162503104+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,164729834+00:00 | gunicorn/run | 2023-02-13T12:48:59,166495459+00:00 | gunicorn/run | 2023-02-13T12:48:59,168390485+00:00 | gunicorn/run | PATH environment variable: /azureml-envs/feast/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2023-02-13T12:48:59,170397713+00:00 | gunicorn/run | PYTHONPATH environment variable: 2023-02-13T12:48:59,171754532+00:00 | gunicorn/run | 2023-02-13T12:48:59,173140651+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting... 2023-02-13T12:48:59,215335234+00:00 - iot-server/finish 1 0 2023-02-13T12:48:59,216607852+00:00 - Exit code 1 is normal. Not restarting iot-server. absl-py==1.4.0 adal==1.2.7 anyio==3.6.2 applicationinsights==0.11.10 argcomplete==2.0.0 async-timeout==4.0.2 attrs==22.2.0 azure-common==1.1.28 azure-core==1.22.1 azure-graphrbac==0.61.1 azure-identity==1.12.0 azure-mgmt-authorization==2.0.0 azure-mgmt-containerregistry==9.1.0 azure-mgmt-core==1.3.2 azure-mgmt-keyvault==9.3.0 azure-mgmt-resource==21.0.0 azure-mgmt-storage==20.0.0 azure-storage-blob==12.14.1 azureml-core==1.42.0.post1 azureml-dataprep==4.0.4 azureml-dataprep-native==38.0.0 azureml-dataprep-rslex==2.6.3 azureml-dataset-runtime==1.42.0 azureml-defaults==1.42.0 azureml-inference-server-http==0.4.14 backports.tempfile==1.0 backports.weakref==1.0.post1 bcrypt==4.0.1 cachetools==5.3.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.15.1 charset-normalizer==3.0.1 click==7.1.2 cloudpickle==2.2.1 colorama==0.4.6 configparser==3.7.4 contextlib2==21.6.0 cryptography==36.0.2 dill==0.3.4 distro==1.8.0 docker==5.0.3 dotnetcore2==3.1.23 fastapi==0.89.1 fastavro==1.7.1 feast==0.18.1 feast-azure-provider==0.3.0 Flask==1.0.3 fusepy==3.0.1 google-api-core==2.11.0 google-auth==2.16.0 googleapis-common-protos==1.52.0 greenlet==2.0.2 grpcio==1.51.1 grpcio-reflection==1.51.1 gunicorn==20.1.0 h11==0.14.0 hiredis==2.2.1 httptools==0.5.0 humanfriendly==10.0 idna==3.4 importlib-resources==5.10.2 inference-schema==1.3.0 isodate==0.6.1 itsdangerous==2.0.1 jeepney==0.8.0 Jinja2==3.0.3 jmespath==1.0.0 joblib==1.1.0 json-logging-py==0.2 jsonpickle==2.2.0 jsonschema==4.17.3 knack==0.9.0 MarkupSafe==2.1.2 mmh3==3.0.0 msal==1.21.0 msal-extensions==1.0.0 msrest==0.6.21 msrestazure==0.6.4 ndg-httpsclient==0.5.1 numpy==1.24.2 oauthlib==3.2.2 opencensus==0.11.1 opencensus-context==0.1.3 opencensus-ext-azure==1.1.8 packaging==21.3 pandas==1.5.3 pandavro==1.5.2 paramiko==2.12.0 pathspec==0.11.0 pkginfo==1.9.6 pkgutil-resolve-name==1.3.10 portalocker==2.7.0 proto-plus==1.19.6 protobuf==3.19.6 psutil==5.9.4 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 pydantic==1.10.4 Pygments==2.14.0 PyJWT==2.6.0 PyNaCl==1.5.0 pyodbc==4.0.35 pyOpenSSL==22.1.0 pyparsing==3.0.9 pyrsistent==0.19.3 PySocks==1.7.1 python-dateutil==2.8.2 python-dotenv==0.21.1 pytz==2022.7.1 PyYAML==6.0 redis==4.4.2 requests==2.28.2 requests-oauthlib==1.3.1 rsa==4.9 scikit-learn==0.22.2.post1 scipy==1.10.0 SecretStorage==3.3.3 six==1.16.0 sniffio==1.3.0 SQLAlchemy==2.0.1 starlette==0.22.0 tabulate==0.8.10 tenacity==8.1.0 tensorflow-metadata==1.12.0 toml==0.10.2 tqdm==4.64.1 typing-extensions==4.4.0 urllib3==1.26.9 uvicorn==0.20.0 uvloop==0.17.0 watchfiles==0.18.1 websocket-client==1.5.1 websockets==10.4 Werkzeug==1.0.1 wrapt==1.12.1 zipp==3.12.1
2023-02-13T12:48:59,515269386+00:00 | gunicorn/run | 2023-02-13T12:48:59,516890508+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,518477530+00:00 | gunicorn/run | AzureML Inference Server 2023-02-13T12:48:59,519997851+00:00 | gunicorn/run | ############################################### 2023-02-13T12:48:59,521348570+00:00 | gunicorn/run | 2023-02-13T12:49:00,337052531+00:00 | gunicorn/run | Starting AzureML Inference Server HTTP.
Azure ML Inferencing HTTP server v0.4.14
Server Settings
Entry Script Name: score.py Model Directory: azureml-models/az_order_model/2 Worker Count: 1 Worker Timeout (seconds): 300 Server Port: 31311 Application Insights Enabled: false Application Insights Key: None
Server Routes
Liveness Probe: GET 127.0.0.1:31311/ Score: POST 127.0.0.1:31311/score
Starting gunicorn 20.1.0 Listening at: http://0.0.0.0:31311/ (13) Using worker: sync Booting worker with pid: 64 Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage Failure while loading azureml_run_type_providers. Failed to load entrypoint azureml.scriptrun = azureml.core.script_run:ScriptRun._from_run_dto with exception (cryptography 36.0.2 (/azureml-envs/feast/lib/python3.8/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'pyopenssl'}). Initializing logger 2023-02-13 12:49:02,162 | root | INFO | Starting up app insights client logging socket was found. logging is available. logging socket was found. logging is available. 2023-02-13 12:49:02,162 | root | INFO | Starting up request id generator 2023-02-13 12:49:02,163 | root | INFO | Starting up app insight hooks 2023-02-13 12:49:02,163 | root | INFO | Invoking user's init function connecting to registry... no request id,connecting to registry...
connecting to repo config... no request id,connecting to repo config...
connecting to feature store... no request id,connecting to feature store...
2023-02-13 12:49:02,174 | root | ERROR | User's init function failed 2023-02-13 12:49:02,179 | root | ERROR | Encountered Exception Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 21, in init from azure.storage.blob import BlobClient File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/init.py", line 10, in
from ._blob_client import BlobClient
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 24, in
from ._shared.base_client import StorageAccountHostsMixin, parse_connection_str, parse_query, TransportWrapper
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 40, in
from .constants import CONNECTION_TIMEOUT, READ_TIMEOUT, SERVICE_HOST_BASE
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/constants.py", line 7, in
from .._serialize import _SUPPORTED_API_VERSIONS
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_serialize.py", line 18, in
from ._models import (
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_models.py", line 18, in
from ._generated.models import ArrowField
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/init.py", line 9, in
from ._azure_blob_storage import AzureBlobStorage
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_azure_blob_storage.py", line 15, in
from . import models
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/init.py", line 9, in
from ._models_py3 import AccessPolicy
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/models/_models_py3.py", line 14, in
from .. import _serialization
File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_serialization.py", line 51, in
from azure.core.exceptions import DeserializationError, SerializationError, raise_with_traceback
ImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 201, in register main.init() File "/var/azureml-app/src/score.py", line 126, in init store = FeatureStore(config=repo_cfg) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 338, in wrapper raise exc.with_traceback(traceback) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 324, in wrapper return func(*args, **kwargs) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/feature_store.py", line 127, in init self._registry = Registry(registry_config, repo_path=self.repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/registry.py", line 166, in init self._registry_store = cls(registry_config, repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 28, in init raise FeastExtrasDependencyImportError("az", str(e)) feast.errors.FeastExtrasDependencyImportError: cannot import name 'DeserializationError' from 'azure.core.exceptions' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/exceptions.py) You may need run pip install 'feast[az]'
2023-02-13 12:49:02,179 | root | INFO | Waiting for logs to be sent to Application Insights before exit. 2023-02-13 12:49:02,179 | root | INFO | Waiting 30 seconds for upload. /azureml-envs/feast/lib/python3.8/site-packages/azureml/data/dataset_type_definitions.py:67: DeprecationWarning: FileType Enum is Deprecated in > 1.0.39. Use strings instead. warnings.warn( Worker exiting (pid: 64) Shutting down: Master Reason: Worker failed to boot. 2023-02-13T12:49:32,454322839+00:00 - gunicorn/finish 3 0 2023-02-13T12:49:32,455683858+00:00 - Exit code 3 is not normal. Killing image.
WebserviceException Traceback (most recent call last) Cell In[4], line 8 4 deployment_config = LocalWebservice.deploy_configuration(port=6789) 6 local_service = Model.deploy(ws, "test", [model], inference_config, deployment_config) ----> 8 local_service.wait_for_deployment()
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/local.py:72, in _in_state..decorator..decorated(self, *args, *kwargs)
69 if self.state not in states:
70 raise WebserviceException('Cannot call {}() when service is {}.'.format(func.name, self.state),
71 logger=module_logger)
---> 72 return func(self, args, **kwargs)
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/core/webservice/local.py:611, in LocalWebservice.wait_for_deployment(self, show_output) 603 """ 604 Poll the running LocalWebservice deployment. 605 (...) 608 :raises: :class:
azureml.exceptions.WebserviceException
609 """ 610 try: --> 611 container_health_check(self._port, 612 self._container, 613 health_url=self._get_health_url(), 614 cleanup_if_failed=False) 616 self.state = LocalWebservice.STATE_RUNNING 618 print('Local webservice is {} at {}'.format(self.state, self._base_url))File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:748, in container_health_check(docker_port, container, health_url, cleanup_if_failed) 745 continue 746 elif container.status == 'exited': 747 # The container has started and crashed. --> 748 _raise_for_container_failure(container, cleanup_if_failed, 749 'Error: Container has crashed. Did your init method fail?') 751 # The container hasn't crashed, so try to ping the health endpoint. 752 try:
File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/azureml/_model_management/_util.py:1265, in _raise_for_container_failure(container, cleanup, message) 1262 if cleanup: 1263 cleanup_container(container) -> 1265 raise WebserviceException(message, logger=module_logger)
WebserviceException: WebserviceException: Message: Error: Container has crashed. Did your init method fail? InnerException None ErrorResponse { "error": { "message": "Error: Container has crashed. Did your init method fail?" } }
SCORE SCRIPT
import os import logging import json import numpy import joblib import pandas as pd from datetime import timedelta
import os import logging import json import joblib from feast import FeatureStore, RepoConfig from feast.registry import RegistryConfig from feast_azure_provider.mssqlserver import MsSqlServerOfflineStoreConfig from feast.infra.online_stores.redis import RedisOnlineStoreConfig, RedisOnlineStore from azureml.core.model import Model from azureml.core import Workspace, Dataset, Datastore
def init(): sql_conn_str = os.getenv("SQL_CONN") redis_conn_str = os.getenv("REDIS_CONN") feast_registry_path = os.getenv("REGISTRY_PATH")
def run(raw_data): data = json.loads(raw_data) feature_vector = store.get_online_features( features=[ "driver_stats:conv_rate", "driver_stats:avg_daily_trips", "driver_stats:acc_rate", "customer_profile:current_balance", "customer_profile:avg_passenger_count", "customer_profile:lifetime_trip_count", ], entity_rows=[data], ).to_df() logging.info(feature_vector) if len(feature_vector.dropna()) > 0: data = feature_vector[ [ "conv_rate", "avg_daily_trips", "acc_rate", "current_balance", "avg_passenger_count", "lifetime_trip_count", ] ]
Note
I have tried different types of deployment, ACI, LOCAL, Managed Online Endpoints in azure but still im facing same problem. i have changed dependencies in docker file same error i have encounter can please suggest where exactly we are missing thanks.