search

keras-team / autokeras

AutoML library for deep learning
http://autokeras.com/
Apache License 2.0
9.12k stars 1.4k forks source link

AutoModel.predict crashes with custom metric #1400

Open perceptualJonathan opened 3 years ago

perceptualJonathan commented 3 years ago

Bug Description

Bug Reproduction

Code for reproducing the bug:

from sklearn.datasets import load_breast_cancer

from tensorflow.keras.callbacks import Callback, EarlyStopping, ModelCheckpoint
from tensorflow.python.platform import tf_logging as logging
from sklearn import datasets, svm, metrics
from sklearn.model_selection import train_test_split, KFold, StratifiedKFold
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, cohen_kappa_score, matthews_corrcoef, log_loss
import matplotlib.pyplot as plt, pandas as pd, numpy as np, matplotlib as mpl
import requests, time, re, os, subprocess, json, sys, datetime

import autokeras as ak
import keras.backend as K
if 0:
    import keras
    #from keras.callbacks import ModelCheckpoint, History, EarlyStopping
    from keras.datasets import fashion_mnist
    from keras.layers import Dense, Dropout, Flatten
    from keras.layers import Conv1D, MaxPooling1D
    from keras.layers.normalization import BatchNormalization
    from keras.layers.advanced_activations import LeakyReLU
    from keras.models import Sequential,Input,Model
    from keras.optimizers import SGD,Adam
    from keras.utils import to_categorical
from IPython.display import clear_output
%matplotlib inline

def matthewsCorrelation(yTrue, yPred):
    yPredPos = K.round(K.clip(yPred, 0, 1))
    yPredNeg = 1 - yPredPos

    yPos = K.round(K.clip(yTrue, 0, 1))
    yNeg = 1 - yPos

    tp = K.sum(yPos * yPredPos) #int(sum(yPos * yPredPos))
    tn = K.sum(yNeg * yPredNeg)

    fp = K.sum(yNeg * yPredPos)
    fn = K.sum(yPos * yPredNeg)

    numerator = (tp * tn - fp * fn)
    denominator = K.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))

    return numerator / (denominator + K.epsilon())

x, y = load_breast_cancer(return_X_y=True)
xTrain, xTest, yTrain, yTest = train_test_split(x, y, random_state=0)

MAXTRIALS = 20
inputNode = ak.StructuredDataInput()
outputNode = ak.StructuredDataBlock(categorical_encoding=True)(inputNode)
outputNode = ak.ClassificationHead(num_classes=2)(outputNode)

clf = ak.AutoModel(
    inputNode, 
    outputNode, 
    overwrite=True,
    objective="val_loss",
    metrics=[matthewsCorrelation],
    max_trials=MAXTRIALS)

clf.fit(xTrain, yTrain) #metrics=[matthewsCorrelation],

predictedClasses = clf.predict(xTest)

cm = confusion_matrix(yTest, predictedClasses)

print(cm)

matthew = matthews_corrcoef(yTest, predictedClasses)

print('MCC:', matthew)

Data used by the code: Breast Cancer Dataset

Expected Behavior

Setup Details

Include the details about the versions of:

Additional context

I have come up with a solution to this problem by editing a couple of the autokeras files. In autokeras/auto_model.py, I changed the predict() function to be 

def predict(self, x, batch_size=32, custom_objects={}, **kwargs):
        """Predict the output for a given testing data.

        # Arguments
            x: Any allowed types according to the input node. Testing data.
            **kwargs: Any arguments supported by keras.Model.predict.

        # Returns
            A list of numpy.ndarray objects or a single numpy.ndarray.
            The predicted results.
        """
        if isinstance(x, tf.data.Dataset):
            if self._has_y(x):
                x = x.map(lambda x, y: x)
        self._check_data_format((x, None), predict=True)
        dataset = self._adapt(x, self.inputs, batch_size)
        pipeline = self.tuner.get_best_pipeline()
        if custom_objects:
            model = self.tuner.get_best_model(custom_objects=custom_objects)
        else:
            model = self.tuner.get_best_model()
        dataset = pipeline.transform_x(dataset)
        y = model.predict(dataset, **kwargs)
        y = utils.predict_with_adaptive_batch_size(
            model=model, batch_size=batch_size, x=dataset, **kwargs
        )
        return pipeline.postprocess(y)

Then in autokeras/engine/tuner.py, I changed the get_best_model() function to be

def get_best_model(self, custom_objects={}):
        with hm_module.maybe_distribute(self.distribution_strategy):
            if custom_objects:
                model = tf.keras.models.load_model(self.best_model_path, custom_objects=custom_objects)
            else:
                model = tf.keras.models.load_model(self.best_model_path)
        return model

Lastly, in the above code that I used, I replace predictedClasses = clf.predict(xTest) with predictedClasses = clf.predict(xTest, custom_objects={'matthewsCorrelation': matthewsCorrelation}). With these changes made, everything runs as I would expect.

garyee commented 3 years ago

Hi,

did get the same error apparently. I got an error message: ValueError: Unable to restore custom object of type _tf_keras_metric currently. Please make sure that the layer implements 'get_config'and 'from_config' when saving. In addition, please use the 'custom_objects' arg when calling 'load_model()'. I used the StructuredDataClassifier. Apparently the custom metric is not saved with the model.

haifeng-jin commented 3 years ago

@garyee Would you paste your code? I am trying to reproduce the issue. We actually did save the custom metric.

garyee commented 3 years ago

https://colab.research.google.com/drive/1wmQx004H-a1QLsOmhmJcw4f4DubjC7qT?usp=sharing

If you uncomment the two parameters in the StructuredDataClassifier definition and run the cell you will get the error. This might be an error I made.

cordeirojoao commented 3 years ago

Hello, having the same issue in here. Any solution to it? Thanks

LucaUrbinati44 commented 3 years ago

I have the same problem.

I think that this line should be changed from this: https://github.com/keras-team/autokeras/blob/0d22c6f6a611cdfc017a24c28c68e7925b7f7feb/autokeras/engine/tuner.py#L63

to something like this: model = tf.keras.models.load_model(self.best_model_path, custom_objects={"custom_metric": custom_metric}) when there is a custom metric provided by the user.

(Source: https://github.com/tensorflow/tensorflow/issues/33648#issuecomment-594908246)

jos1977 commented 3 years ago

Hi, i can confirm this issue is still present in the latest Autokeras version: v1.0.12. In my case i found out when i wanted to to get the best model after using a StructuredDataClassifier (using custom metrics loss function) with:

# get the best performing model best_model = reg.export_model()

Which ended up with the following error:

ValueError: Unable to restore custom object of type _tf_keras_metric currently. Please make sure that the layer implements 'get_config'and 'from_config' when saving. In addition, please use the 'custom_objects' arg when calling 'load_model()'.

I managed to fix this issue with the solution of @KeikiHekili mentioned earlier (the changes in autokeras/auto_model.py and autokeras/autokeras/engine/tuner.py ) and also changing the following in autokeras/auto_model.py

Original:

    def export_model(self):
        """Export the best Keras Model.

        # Returns
            tf.keras.Model instance. The best model found during the search, loaded
            with trained weights.
        """
        return self.tuner.get_best_model()

Changed:

    def export_model(self, custom_objects={}):
        """Export the best Keras Model.

        # Returns
            tf.keras.Model instance. The best model found during the search, loaded
            with trained weights.
        """
        if custom_objects:
            return self.tuner.get_best_model(custom_objects=custom_objects)
        else:
            return self.tuner.get_best_model()

After this change i could use a custom metric as follow:

import kerastuner
reg = ak.StructuredDataRegressor(max_trials=3, overwrite=True, metrics=[spearman_rankcor],objective=kerastuner.Objective('spearman_rankcor', direction='max'))
# Feed the structured data regressor with training data.
reg.fit(training_data[feature_names], training_data[TARGET_NAME], epochs=10)

I performed the model export and saving with:

# get the best performing model
best_model = reg.export_model(custom_objects={'spearman_rankcor': spearman_rankcor})
# summarize the loaded model
best_model.summary()
# Now save the model with round number
logging.info("saving model: %s", MODEL_FILE)
best_model.save(MODEL_FILE)

I still need to validate the actual model and if custom metric is used properly but so far the logging output looks ok.

zuliani99 commented 3 years ago

I also notice this problem with the evaluation method. So I tried to solve it the same way you did. But this error returns:

/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1323 test_function  *
    return step_function(self, iterator)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1314 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1285 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2833 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3608 _call_for_each_replica
     return fn(*args, **kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1309 run_step  **
    with ops.control_dependencies(_minimum_control_deps(outputs)):
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:2888 _minimum_control_deps
    outputs = nest.flatten(outputs, expand_composites=True)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/util/nest.py:416 flatten
    return _pywrap_utils.Flatten(structure, expand_composites)

TypeError: '<' not supported between instances of 'function' and 'str'
JuliaWasala commented 2 years ago

I also notice this problem with the evaluation method. So I tried to solve it the same way you did. But this error returns:

/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1323 test_function  *
    return step_function(self, iterator)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1314 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1285 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2833 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3608 _call_for_each_replica
     return fn(*args, **kwargs)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1309 run_step  **
    with ops.control_dependencies(_minimum_control_deps(outputs)):
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:2888 _minimum_control_deps
    outputs = nest.flatten(outputs, expand_composites=True)
/home/riccardo/Desktop/venv/lib/python3.8/site-packages/tensorflow/python/util/nest.py:416 flatten
    return _pywrap_utils.Flatten(structure, expand_composites)

TypeError: '<' not supported between instances of 'function' and 'str'

This problem is still going on. This post https://stackoverflow.com/questions/65549053/typeerror-not-supported-between-instances-of-function-and-str says that it can be fixed by compiling the model, but automodel doesnt have direct access to the loss and optimizer. I wonder whether compiling without it can be fixed. I'm going to try it out. It would be great if this issue could be fixed, it's been going on a long time. Either that, or maybe remove from the documentation that custom metrics can be used, because it makes it look like it's simple to use but at the moment it is not

JuliaWasala commented 2 years ago

I can confirm that the combination of the other edits suggested in this issue, with the addition of compiling the model with the custom metric in evaluate solves the issue. For me it works with the following version of autoModel.evaluate:

    def evaluate(self, x, y=None, batch_size=32, verbose=1, custom_objects={},**kwargs):
        """Evaluate the best model for the given data.

        # Arguments
            x: Any allowed types according to the input node. Testing data.
            y: Any allowed types according to the head. Testing targets.
                Defaults to None.
            batch_size: Number of samples per batch.
                If unspecified, batch_size will default to 32.
            verbose: Verbosity mode. 0 = silent, 1 = progress bar.
                Controls the verbosity of
                [keras.Model.evaluate](http://tensorflow.org/api_docs/python/tf/keras/Model#evaluate)
            **kwargs: Any arguments supported by keras.Model.evaluate.

        # Returns
            Scalar test loss (if the model has a single output and no metrics) or
            list of scalars (if the model has multiple outputs and/or metrics).
            The attribute model.metrics_names will give you the display labels for
            the scalar outputs.
        """
        self._check_data_format((x, y))
        if isinstance(x, tf.data.Dataset):
            dataset = x
            x = dataset.map(lambda x, y: x)
            y = dataset.map(lambda x, y: y)
        x = self._adapt(x, self.inputs, batch_size)
        y = self._adapt(y, self._heads, batch_size)
        dataset = tf.data.Dataset.zip((x, y))
        pipeline = self.tuner.get_best_pipeline()
        dataset = pipeline.transform(dataset)
        if custom_objects:
            model = self.tuner.get_best_model(custom_objects=custom_objects)
            # only gets metrics from custom_objects for now
            model.compile(metrics=[val for key,val in custom_objects.items()])
        else:
            model = self.tuner.get_best_model()
        return utils.evaluate_with_adaptive_batch_size(
            model=model, batch_size=batch_size, x=dataset, verbose=verbose, **kwargs

I compile the model with only the metrics provided in custom objects.