Open NKUCodingCat opened 5 years ago
I think I found the reason but I have no idea about how to fix it
estimator.model_to_estimator will replace the graph built by keras with the graph they make, and the class is stateful, so an object, for example, built by true_positive, keeps an variant as self.tp, which is built by keras when I call it, but the graph of variant y_true/y_pred is replaced
I am not sure why there is a stateful layer because I check all loss/metrics of keras, I haven't found any loss is relies on the state of object(i.e. they are stateless), but I am not sure how keras_metrics work... Could you please provide some advices @ybubnov ?
Seems it is caused by the behaviour of when model_to_estimator process layer and metrics, it might replace the graph of any layer(I guess) but it will not replace the graph in metrics since they assumed that the metrics is STATLESS.
As far as I concerned, true_positive is implemented as a layer will reduce the cost of calculation when user require multiple metrics provide by keras_metrics(? I am not sure, I had been confused by the calling chain). IF what I think is right, I think it is a meaningless optimization because it makes potential incompatibles.
Sorry for my broken english and hope it helps
I also faced a samliar issue; the Metrics does not work with tf.keras.estimator.model_to_estimator. if I remove the tf.keras.mertics.AUC the code can work. or else it has an error like this:
return fn(*args, **kwargs)
File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_utils.py", line 873, in call_metric_function return metric_fn(y_true, y_pred, sample_weight=weights) File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/keras/metrics.py", line 170, in call update_op = self.update_state(*args, *kwargs) # pylint: disable=not-callable File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/keras/utils/metrics_utils.py", line 73, in decorated update_op = update_state_fn(args, **kwargs) File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/keras/metrics.py", line 1715, in update_state }, y_true, y_pred, self.thresholds, sample_weight=sample_weight) File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/keras/utils/metrics_utils.py", line 268, in update_confusion_matrix_variables y_pred.shape.assert_is_compatible_with(y_true.shape) File "/Users/sangyongjia/anaconda3/envs/tf1.x/lib/python3.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 1103, in assert_is_compatible_with raise ValueError("Shapes %s and %s are incompatible" % (self, other)) ValueError: Shapes (?, 1) and (?,) are incompatible
the correspond code is : model.compile("adam", "binary_crossentropy", metrics=[tf.keras.metrics.BinaryCrossentropy(),tf.keras.metrics.AUC()])
if I remove tf.keras.metrics.AUC() model.compile("adam", "binary_crossentropy", metrics=[tf.keras.metrics.BinaryCrossentropy()]) it can work
anyone has some great idea? thx in advance.
@sangyongjia, it looks that your problem relates to TensorFlow project, not to the keras-metrics
@sangyongjia, it looks that your problem relates to TensorFlow project, not to the
keras-metrics
you mean I should remove the Tensorflow and install it again or update to higher version of tf ? Now the version of tf I am using is 1.14。
@ybubnov
@sangyongjia, it looks that the problem is in the TensorFlow
itself, particularly in to_model_estimator
call, JosPolfliet created an issue in that project a long time ago: https://github.com/tensorflow/tensorflow/issues/34040 and it is still unresolved.
I think one of possible ways to deal with the problem is to enforce issue resolution I've posted above. Alternatively, you could not use model_to_estimator
if that is possible (simply use a model).
From my side, I'll try to relax the TensorFlow restriction in order to allow newer version of TensorFlow be used with keras-metrics
library.
thx so much @ybubnov
I am trying to use
tf.keras.estimator.model_to_estimator
to convert tf.keras model to be distributed, however, I found that keras-metrics does not work as desired, Is there any idea or work around for me ? thanksTraceback:
Buggy code (a little bit messy...)
model.py.zip
If I dont add
keras_metrics.sparse_categorical_precision()
into Accuracy part, it DOES work but fail when I add sparse_categorical_precision...Tested in Py2.7/3.7 TF 1.13.1