Thank you for submitting an issue. Please refer to our issue policy for additional information about bug reports. For help with debugging your code, please refer to Stack Overflow.
Please fill in this bug report template to ensure a timely and thorough response.
Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?
[ ] Yes. I can contribute a fix for this bug independently.
[ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
[ X] No. I cannot contribute a bug fix at this time.
System information
Have I written custom code (as opposed to using a stock example script provided in MLflow): Custom
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Big Sur 11.2.3
MLflow installed from (source or binary): binary
MLflow version (run mlflow --version): 1.15.0
Python version: 3.8.8
npm version, if running the dev UI:
Exact command to reproduce:
Describe the problem
Actual behavior:
When a TensorFlow model with a signature is serialized and used for a prediction the MLflow predict method throws an exception.
Expected behavior:mlflow_model.predict(feed_dict) shouldn't throw an exception when using a signature.
Code to reproduce issue
import unittest
import mlflow
import numpy as np
import tensorflow as tf
from mlflow.tensorflow import _TF2Wrapper
class TestTfModelPredict(unittest.TestCase):
def test_tf_model_predict(self):
tf.random.set_seed(1337)
TF_MODEL_DIR = "model"
# Build model
inputs = tf.keras.layers.Input(shape=1, name='input', dtype=tf.float32)
outputs = tf.keras.layers.Dense(1)(inputs)
model = tf.keras.Model(inputs=inputs, outputs=[outputs])
# Compile model
model.compile(
optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
# Serving signature
@tf.function(input_signature=[tf.TensorSpec(shape=[1, 1], dtype=tf.float32, name='input')])
def serve_fn(example):
outputs = model(example)
return outputs
# Save model with serving signature
tf.saved_model.save(model, TF_MODEL_DIR,
signatures={'serving_default': serve_fn.get_concrete_function()})
feed_dict = {"input": tf.constant([[2.0]])}
loaded_model = tf.saved_model.load(TF_MODEL_DIR)
infer = loaded_model.signatures['serving_default']
prediction = infer(**feed_dict)
# This works as expected
np.testing.assert_allclose(prediction['output_0'], np.asarray([[-0.09599352]]))
# Save as mlflow model
with mlflow.start_run() as run:
mlflow.tensorflow.log_model(
tf_saved_model_dir=TF_MODEL_DIR,
tf_meta_graph_tags=['serve'],
tf_signature_def_key='serving_default',
artifact_path='mlflow_model'
)
loaded_model = tf.saved_model.load(
export_dir=f'mlruns/0/{run.info.run_id}/artifacts/mlflow_model/tfmodel',
tags=['serve']
)
tf2_wrapper = _TF2Wrapper(infer=loaded_model.signatures['serving_default'])
prediction = tf2_wrapper.predict(feed_dict)
# This works as expected
np.testing.assert_allclose(prediction['output_0'], np.asarray([-0.09599352]))
model_uri = f'mlruns/0/{run.info.run_id}/artifacts/mlflow_model'
mlflow_model = mlflow.pyfunc.load_model(model_uri)
# This doesn't work (!!!)
prediction = mlflow_model.predict(feed_dict)
np.testing.assert_allclose(prediction['output_0'], np.asarray([-0.09599352]))
if __name__ == '__main__':
unittest.main()
Other info / logs
Error
Traceback (most recent call last):
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 60, in testPartExecutor
yield
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
method()
File "/Users/schramm/PycharmProjects/kiwi/test-mlflow2/tensorflow/test.py", line 62, in test_tf_model_predict
prediction = mlflow_model.predict(feed_dict)
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/mlflow/pyfunc/init.py", line 575, in predict
return self._model_impl.predict(data)
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/mlflow/tensorflow.py", line 582, in predict
raw_preds = self.infer(**feed_dict)
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1669, in call
return self._call_impl(args, kwargs)
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1678, in _call_impl
return self._call_with_structured_signature(args, kwargs,
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1759, in _call_with_structured_signature
return self._call_flat(
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 115, in _call_flat
return super(_WrapperFunction, self)._call_flat(args, captured_inputs,
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call
outputs = execute.execute(
File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable _AnonymousVar15 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar15/N10tensorflow3VarE does not exist.
[[{{node StatefulPartitionedCall/model/dense/BiasAdd/ReadVariableOp}}]] [Op:__inference_signature_wrapper_932]
Function call stack:
signature_wrapper
What component(s), interfaces, languages, and integrations does this bug affect?
Components
[ ] area/artifacts: Artifact stores and artifact logging
[ ] area/build: Build and test infrastructure for MLflow
[ ] area/docs: MLflow documentation pages
[ ] area/examples: Example code
[ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
[X ] area/models: MLmodel format, model serialization/deserialization, flavors
[ ] area/projects: MLproject format, project running backends
[ X] area/scoring: Local serving, model deployment tools, spark UDFs
[ ] area/server-infra: MLflow server, JavaScript dev server
Thank you for submitting an issue. Please refer to our issue policy for additional information about bug reports. For help with debugging your code, please refer to Stack Overflow.
Please fill in this bug report template to ensure a timely and thorough response.
Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?
System information
mlflow --version
): 1.15.0Describe the problem
Actual behavior:
When a TensorFlow model with a signature is serialized and used for a prediction the MLflow predict method throws an exception.
Model with signature being saved (this works):
This throws an exception:
Interestingly an exception is not thrown when the TF2Wrapper is instantiated directly:
Expected behavior:
mlflow_model.predict(feed_dict)
shouldn't throw an exception when using a signature.Code to reproduce issue
Other info / logs
Error Traceback (most recent call last): File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 60, in testPartExecutor yield File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 676, in run self._callTestMethod(testMethod) File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/unittest/case.py", line 633, in _callTestMethod method() File "/Users/schramm/PycharmProjects/kiwi/test-mlflow2/tensorflow/test.py", line 62, in test_tf_model_predict prediction = mlflow_model.predict(feed_dict) File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/mlflow/pyfunc/init.py", line 575, in predict return self._model_impl.predict(data) File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/mlflow/tensorflow.py", line 582, in predict raw_preds = self.infer(**feed_dict) File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1669, in call return self._call_impl(args, kwargs) File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1678, in _call_impl return self._call_with_structured_signature(args, kwargs, File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1759, in _call_with_structured_signature return self._call_flat( File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 115, in _call_flat return super(_WrapperFunction, self)._call_flat(args, captured_inputs, File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1918, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 555, in call outputs = execute.execute( File "/usr/local/anaconda3/envs/test-mlflow2/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable _AnonymousVar15 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar15/N10tensorflow3VarE does not exist. [[{{node StatefulPartitionedCall/model/dense/BiasAdd/ReadVariableOp}}]] [Op:__inference_signature_wrapper_932]
Function call stack: signature_wrapper
What component(s), interfaces, languages, and integrations does this bug affect?
Components
area/artifacts
: Artifact stores and artifact loggingarea/build
: Build and test infrastructure for MLflowarea/docs
: MLflow documentation pagesarea/examples
: Example codearea/model-registry
: Model Registry service, APIs, and the fluent client calls for Model Registryarea/models
: MLmodel format, model serialization/deserialization, flavorsarea/projects
: MLproject format, project running backendsarea/scoring
: Local serving, model deployment tools, spark UDFsarea/server-infra
: MLflow server, JavaScript dev serverarea/tracking
: Tracking Service, tracking client APIs, autologgingInterface
area/uiux
: Front-end, user experience, JavaScript, plottingarea/docker
: Docker use across MLflow's components, such as MLflow Projects and MLflow Modelsarea/sqlalchemy
: Use of SQLAlchemy in the Tracking Service or Model Registryarea/windows
: Windows supportLanguage
language/r
: R APIs and clientslanguage/java
: Java APIs and clientslanguage/new
: Proposals for new client languagesIntegrations
integrations/azure
: Azure and Azure ML integrationsintegrations/sagemaker
: SageMaker integrationsintegrations/databricks
: Databricks integrations