Unable to Successfully Consume Inference Endpoint

nhwkuhns commented 3 years ago

[Enter feedback here] I appears the code within the notebook 02-turbofan_deploy_model.ipynb has some issues both with creating the container image and getting predictions once the model is deployed.

First, I could not get the image creation to work with the myenv.yml. I had to use the docker_file parameter with the below entries to get everything to deploy correctly.


RUN apt-get update
RUN apt-get install -y build-essential python-software-properties software-properties-common gcc

RUN conda install -c r -y pip=20.1.1 && \
  conda install -c conda-forge gxx_linux-64 scikit-learn numpy pandas psutil gcc && \
    conda clean -ay

RUN pip install --no-cache-dir azureml-defaults azureml-sdk azureml-automl-core azureml-automl-dnn-nlp azureml-automl-runtime scikit-learn==0.22.1 xgboost==0.90 plotly

Second, once the model gets deployed to ACI, I get the following error:

{ "error": { "code": "UserError", "message": "Expected column(s) 0 not found in fitted data.", "target": "X", "inner_error": { "code": "BadArgument", "inner_error": { "code": "MissingColumnsInData" } }, "reference_code": "17049f70-3bbe-4060-a63f-f06590e784e5" } }

I believed the error to be caused by the training and test dataframes having an unnamed column left over from not having "index=False" in the "to_csv()" method. Upon modifying the code in 01-turbofan_regression.ipynb and retraining the model, I still get the issue.

The Image and ContainerImage classes are deprecated, which may be creating some of these issues. I am working with a customer to fix these issues, but have not had luck in the past couple days.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: de25b89e-91d8-3df4-c2e8-a98060e0c9b2
Version Independent ID: 55403f3e-4c47-63c7-310c-bd9374c93015
Content: Tutorial: Train and deploy a model - Machine Learning on Azure IoT Edge
Content Source: articles/iot-edge/tutorial-machine-learning-edge-04-train-model.md
Service: iot-edge
GitHub Login: @kgremban
Microsoft Alias: kgremban

SatishBoddu-MSFT commented 3 years ago

Hello @nhwkuhns We are looking into this.

nhwkuhns commented 3 years ago

I was able to pinpoint the issue. Turns out the score.py script was sending the dataframe for inferencing as a np.array.

Also, I was able to build and deploy the model using the Environment class. The code in the 02-....ipynb should be updated to include that rather than the ContainerImage class as the latter is deprecated and it does not cache the image for rebuilding on minor code changes (saving roughly 15 minutes per rebuild for code changes).

SatishBoddu-MSFT commented 2 years ago

Hello @nhwkuhns Is this the right Github : https://github.com/Azure-Samples/IoTEdgeAndMlSample/tree/master/AzureNotebooks ?

For the Notebook code sample files?

MicrosoftDocs / azure-docs

Unable to Successfully Consume Inference Endpoint #80849

Document Details