Example deployment fails

[Ritaja]

train and deploy with Feast notebook fails (Part 3). When the following block is executed:

import uuid
from azureml.core.model import InferenceConfig
from azureml.core.environment import Environment
from azureml.core.model import Model

# get the registered model
model = Model(ws, "order_model")

# create an inference config i.e. the scoring script and environment
inference_config = InferenceConfig(
    entry_script="score.py", 
    environment=env, 
    source_directory="src"
)

# deploy the service
service_name = "orders-service" + str(uuid.uuid4())[:4]
service = Model.deploy(
    workspace=ws,
    name=service_name,
    models=[model],
    inference_config=inference_config,
    deployment_config=aciconfig,
)

service.wait_for_deployment(show_output=True)

The error is: ImportError: cannot import name 'case_insensitive_dict' from 'azure.core.utils'

Longer trace: Error:

{
  "code": "AciDeploymentFailed",
  "statusCode": 400,
  "message": "Aci Deployment failed with exception: Error in entry script, ImportError: cannot import name 'case_insensitive_dict' from 'azure.core.utils' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/utils/__init__.py), please run print(service.get_logs()) to get details.",
  "details": [
    {
      "code": "CrashLoopBackOff",
      "message": "Error in entry script, ImportError: cannot import name 'case_insensitive_dict' from 'azure.core.utils' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/utils/__init__.py), please run print(service.get_logs()) to get details."
    }
  ]
}

[l-sudarsan]

Gettting same error in aci deployment.

Tried local deployment and getting a similar error:

2022-08-07T05:03:22,187381485+00:00 | gunicorn/run | 2022-08-07T05:03:22,189075413+00:00 | gunicorn/run | ############################################### 2022-08-07T05:03:22,190729342+00:00 | gunicorn/run | AzureML Inference Server 2022-08-07T05:03:22,192228167+00:00 | gunicorn/run | ############################################### 2022-08-07T05:03:22,193847795+00:00 | gunicorn/run | 2022-08-07T05:03:23,119952197+00:00 | gunicorn/run | Starting AzureML Inference Server HTTP.

Azure ML Inferencing HTTP server v0.4.14

Server Settings

Entry Script Name: score.py Model Directory: azureml-models/order_model/5 Worker Count: 1 Worker Timeout (seconds): 300 Server Port: 31311 Application Insights Enabled: false Application Insights Key: None

Server Routes

Liveness Probe: GET 127.0.0.1:31311/ Score: POST 127.0.0.1:31311/score

Starting gunicorn 20.1.0 Listening at: http://0.0.0.0:31311/ (23) Using worker: sync Booting worker with pid: 74 Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage Initializing logger 2022-08-07 05:03:24,355 | root | INFO | Starting up app insights client logging socket was found. logging is available. logging socket was found. logging is available. 2022-08-07 05:03:24,355 | root | INFO | Starting up request id generator 2022-08-07 05:03:24,355 | root | INFO | Starting up app insight hooks 2022-08-07 05:03:24,356 | root | INFO | Invoking user's init function connecting to registry... no request id,connecting to registry...

connecting to repo config... no request id,connecting to repo config...

connecting to feature store... no request id,connecting to feature store...

2022-08-07 05:03:24,463 | root | ERROR | User's init function failed 2022-08-07 05:03:24,468 | root | ERROR | Encountered Exception Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 21, in init from azure.storage.blob import BlobClient File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/init.py", line 10, in from ._blob_client import BlobClient File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 24, in from ._shared.base_client import StorageAccountHostsMixin, parse_connection_str, parse_query, TransportWrapper File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 37, in from .constants import CONNECTION_TIMEOUT, READ_TIMEOUT, SERVICE_HOST_BASE File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_shared/constants.py", line 7, in from .._serialize import _SUPPORTED_API_VERSIONS File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_serialize.py", line 18, in from ._models import ( File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_models.py", line 18, in from ._generated.models import ArrowField File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/init.py", line 9, in from ._azure_blob_storage import AzureBlobStorage File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/_azure_blob_storage.py", line 18, in from .operations import AppendBlobOperations, BlobOperations, BlockBlobOperations, ContainerOperations, PageBlobOperations, ServiceOperations File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/init.py", line 9, in from ._service_operations import ServiceOperations File "/azureml-envs/feast/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_service_operations.py", line 18, in from azure.core.utils import case_insensitive_dict ImportError: cannot import name 'case_insensitive_dict' from 'azure.core.utils' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/utils/init.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/azureml-envs/feast/lib/python3.8/site-packages/azureml_inference_server_http/server/aml_blueprint.py", line 201, in register main.init() File "/var/azureml-app/src/score.py", line 35, in init store = FeatureStore(config=repo_cfg) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 338, in wrapper raise exc.with_traceback(traceback) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/usage.py", line 324, in wrapper return func(*args, **kwargs) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/feature_store.py", line 127, in init self._registry = Registry(registry_config, repo_path=self.repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast/registry.py", line 166, in init self._registry_store = cls(registry_config, repo_path) File "/azureml-envs/feast/lib/python3.8/site-packages/feast_azure_provider/registry_store.py", line 28, in init raise FeastExtrasDependencyImportError("az", str(e)) feast.errors.FeastExtrasDependencyImportError: cannot import name 'case_insensitive_dict' from 'azure.core.utils' (/azureml-envs/feast/lib/python3.8/site-packages/azure/core/utils/init.py) You may need run pip install 'feast[az]'

[gaya3kris]

We are facing the same issue and would like to know how to resolve it.

[aroramaxus]

hey @Ritaja @l-sudarsan @ttthree @sagarsumant , we (@gaya3kris and I) fixed this issue making sure that all version are set to latest in inference.dockerfile. E.g. below:

RUN pip install \ 'protobuf>=3.10,<3.20' \ 'azureml-defaults==1.45.0' \ 'feast-azure-provider==0.3.0' \ 'scikit-learn==1.1' \ 'joblib===1.1.0' \ 'itsdangerous==2.1.2'

the link to the file is https://github.com/Azure/feast-azure/blob/main/provider/tutorial/notebooks/inference.dockerfile

see if this works. Along with this, I also upgraded the az cli version to latest on the compute that was running this code. Plus, also made sure that we are importing joblib in the python code where required.