Trouble Registering a Tensorflow Model

[faradawn]

Hi,

I have been reading the "Example_ExternalModels.ipynb" and wanted to register a Tensorflow keras model.

However, after I converted the Keras model into an Estimator (please correct me if this is not right), the model registering step failed saying "'generator' object has no attribute 'shape'".

Below was the code:

# https://www.tensorflow.org/tutorials/estimator/keras_model_to_estimator
import tempfile
model_dir = tempfile.mkdtemp()

normalizer = layers.Normalization(axis=-1)
# normalizer.adapt(np.array(train_x)) # this causes model_to_estimator to fail

dnn_model = keras.Sequential([
        normalizer,
        layers.Dense(512, activation='relu', input_dim=train_x.shape[1]),
        layers.Dense(512, activation='relu'),
        layers.Dense(1, activation='sigmoid')
])

dnn_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])

keras_estimator = tf.keras.estimator.model_to_estimator(keras_model=dnn_model, model_dir=model_dir)

from piml import Experiment
exp_2 = Experiment()
pipeline_1 = exp_2.make_pipeline(model=keras_estimator, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp_2.register(pipeline_1, "my-dnn")

Here is the error:

AttributeError                            Traceback (most recent call last)
[<ipython-input-34-505e7d52087b>](https://localhost:8080/#) in <module>
     22                              test_x=test_x, test_y=test_y.ravel(),
     23                              feature_names=feature_names, target_name="latency")
---> 24 exp_2.register(pipeline_1, "my-dnn")

1 frames
/usr/local/lib/python3.9/dist-packages/piml/workflow/base.cpython-39-x86_64-linux-gnu.so in piml.workflow.base.Model.register_model()

AttributeError: 'generator' object has no attribute 'shape'

I was running on Google Colab with the installation of

!pip install piml
!pip install numpy==1.23.5

For train and test data, I used sklearn's standard train_test_split. And make_pipline() seemed successful.

Appreciate any help on how to register a Tensorflow model.

Thanks in advance!

[ZebinYang]

Hi @faradawn

PiML only supports sklearn style models. You can wrap your tensorflow models by following codes.

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier

model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

After that, you will be able to do model_explain and model_diagnose. However, this model cannot be interpreted by exp.model_interpret, as it is not a built-in model of PiML.

[faradawn]

Hi Zebin,

Thank you for suggesting a scikit_learn wrapper!

normalizer = layers.Normalization()
# normalizer.adapt(np.array(train_x)) # this causes model_to_estimator to fail
dnn_model = keras.Sequential([
        normalizer,
        layers.Dense(512, activation='relu', input_dim=train_x.shape[1]),
        layers.Dense(512, activation='relu'),
        layers.Dense(1, activation='sigmoid')
])

dnn_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

from piml import Experiment
exp = Experiment()

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

I tried but received a normalization layer problem:

ValueError                                Traceback (most recent call last)
[<ipython-input-8-99a663406776>](https://localhost:8080/#) in <cell line: 23>()
     21                              test_x=test_x, test_y=test_y.ravel(),
     22                              feature_names=feature_names, target_name="latency")
---> 23 exp.register(pipeline_1, "my-dnn")
     24 
     25 

4 frames
[/usr/local/lib/python3.9/dist-packages/keras/engine/training.py](https://localhost:8080/#) in tf__predict_function(iterator)
     13                 try:
     14                     do_return = True
---> 15                     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
     16                 except:
     17                     do_return = False

ValueError: in user code:

    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2169, in predict_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2155, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2143, in run_step  **
        outputs = model.predict_step(data)
    File "/usr/local/lib/python3.9/dist-packages/keras/engine/training.py", line 2111, in predict_step
        return self(x, training=False)
    File "/usr/local/lib/python3.9/dist-packages/keras/utils/traceback_utils.py", line 70, in error_handler
        raise e.with_traceback(filtered_tb) from None

    ValueError: Exception encountered when calling layer 'normalization_4' (type Normalization).

    Dimensions must be equal, but are 8 and 9 for '{{node sequential_3/normalization_4/sub}} = Sub[T=DT_FLOAT](IteratorGetNext, sequential_3/normalization_4/sub/y)' with input shapes: [32,8], [1,9].

    Call arguments received by layer 'normalization_4' (type Normalization):
      • inputs=tf.Tensor(shape=(32, 8), dtype=float32)

I can try to debug more. But the method you suggested was making progress!

Thank you very much, and will let you know if the issue is solved!

[faradawn]

Hi, The tensorflow.keras.wrappers.scikit_learn wrapper allowed me to register a TensorFlow model! After removing the normalization layer, the problem above was resolved.

dnn_model = keras.Sequential([
        layers.Dense(512, activation='relu', input_dim=train_x.shape[1]),
        layers.Dense(512, activation='relu'),
        layers.Dense(1, activation='sigmoid')
])

dnn_model.compile(loss='binary_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])

from tensorflow.keras.wrappers.scikit_learn import KerasClassifier
model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
model.fit(train_x, train_y)
model._estimator_type = "classifier"

from piml import Experiment
exp = Experiment()

pipeline_1 = exp.make_pipeline(model=model, train_x=train_x, train_y=train_y.ravel(),
                             test_x=test_x, test_y=test_y.ravel(),
                             feature_names=feature_names, target_name="latency")
exp.register(pipeline_1, "my-dnn")

And the output is

  model = KerasClassifier(build_fn = lambda : dnn_model, verbose=False)
156/156 [==============================] - 0s 2ms/step
156/156 [==============================] - 0s 3ms/step
Register my-dnn Done

May I ask two more questions: 1) Why global-explainability is not showing?

2) Is there a way to silence the training output during exp.model_explain()?

I read that we can set model.fit(verbose = 0) during training. But how to specify this parameter when PIML calls this model?

Thanks!

[ZebinYang]

Hi @faradawn,

It seems that the verbose parameter in KerasClassifier is not passed to its predict function. To turn off the messages, a workaround is to override the predict functions manually, see below.

def wrap_predict(x):
    proba = model.model.predict(x, verbose=False)
    if proba.shape[-1] > 1:
        classes = proba.argmax(axis=-1)
    else:
        classes = (proba > 0.5).astype("int32")
    return model.classes_[classes]

def wrap_predict_proba(x):
    probs = model.model.predict(x, verbose=False)
    # check if binary classification
    if probs.shape[1] == 1:
        # first column is probability of class 0 and second is of class 1
        probs = np.hstack([1 - probs, probs])
    return probs

model.predict = wrap_predict
model.predict_proba = wrap_predict_proba

The explainability output can be empty if the model output is constant. I tried on my side the built-in 'CoCircles' dataset and the output looks good. Can you double-check the data and model you used? Or just put a reproducible Colab link here.

from piml import Experiment

exp = Experiment()
exp.data_loader("CoCircles", silent=True)
exp.data_summary(silent=True)
exp.data_prepare(silent=True)
feature_names = exp.get_feature_names()
train_x, train_y, train_sample_weight = exp.get_data(train=True)
test_x, test_y, test_sample_weight = exp.get_data(test=True)

[faradawn]

Hi Zebin,

The wrap_predict worked perfectly in that it suppressed the training output!

Regarding the empty permutation feature importance, I created a Colab Notebook that produces the empty graph.

I re-checked the dataset and the model which uses 8 features to make a 0/1 binary prediction.

You already helped so much. Truly appreciate any guidance!

Thanks!

[ZebinYang]

Hi @faradawn

It seems your response variable only has one class

Also, for neural network models, we usually standardize x before model training.

[faradawn]

Hi Zebin,

Got that it was due to y having a single class! Thanks for finding that out!

The issue is resolved. Appreciate your numerous help, such as teaching how to wrap a tensor flow model into skilearn!

Can close issue anytime!