Tensorflow 2 throws OperatorNotAllowedInGraphError in Example code

Bill-Fujimoto commented 4 years ago

Thank you very much for reporting a bug on Talos. Before you do, please go through the below checklist carefully and make sure to prepare your bug report in a way that facilitates effective handling of the matter.

1) Confirm the below

[x] My Python version is 3.5 or higher
[x] I have searched through the issues Issues for a duplicate
[x] I've tested that my Keras model works as a stand-alone

2) Include the output of:

talos.__version__ 0.6.4

3) Explain clearly what you expect to happen

When running Hyperparameter Optimization on Keras with Breast Cancer Data.ipynb example file in Tensorflow 2 conda environment, the following error is thrown: OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

4) Explain what actually happened

This file works OK in Tensorflow 1.15 conda environment

5) Provide a code-complete reference

See attached example notebook with error traceback included. Hyperparameter Optimization on Keras with Breast Cancer Data.zip

[ ] My bug report includes an input model
[x] My bug report includes a parameter dictionary
[x] My bug report includes a Scan() command
[ ] My bug report question includes a link to a sample of the data

NOTE: If the data is sensitive and can't be shared, create dummy data that mimics it.

A self-contained Jupyter Notebook, Google Colab, or similar is highly preferred and will speed up helping you with your issue.

github-actions[bot] commented 4 years ago

Welcome to Talos community! Thanks so much for creating your first issue :)

mikkokotila commented 4 years ago

Sorry for taking ages to come to this. Can you share the whole trace, I'll try to sort this out promptly once you do.

Bill-Fujimoto commented 4 years ago

The trace is already included, see the attached notebook zip file.

mikkokotila commented 4 years ago

It seems to be an open issue in tf.

mikkokotila commented 4 years ago

Removing f1score from the input model will resolve this specific case.

mikkokotila commented 4 years ago

Related with TensorFlow >=2.0 #451

wgeul commented 4 years ago

Traceback (most recent call last):

File "", line 1, in scans = c.gather(scans)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/distributed/client.py", line 1894, in gather asynchronous=asynchronous,

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/distributed/client.py", line 778, in sync self.loop, func, *args, callback_timeout=callback_timeout, **kwargs

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync raise exc.with_traceback(tb)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/distributed/utils.py", line 332, in f result[0] = yield future

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tornado/gen.py", line 735, in run value = future.result()

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/distributed/client.py", line 1753, in _gather raise exception.with_traceback(traceback)

File "", line 17, in process_scan disable_progress_bar=True)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/scan/Scan.py", line 196, in init scan_run(self)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/scan/scan_run.py", line 26, in scan_run self = scan_round(self)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/scan/scan_round.py", line 19, in scan_round self.model_history, self.round_model = ingest_model(self)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/model/ingest_model.py", line 10, in ingest_model self.round_params)

File "", line 17, in model metrics=['acc', talos.utils.metrics.f1score])

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/training/tracking/base.py", line 457, in _method_wrapper result = method(self, *args, **kwargs)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 439, in compile masks=self._prepare_output_masks())

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 2004, in _handle_metrics target, output, output_mask))

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 1955, in _handle_per_output_metrics metric_fn, y_true, y_pred, weights=weights, mask=mask)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_utils.py", line 1155, in call_metric_function return metric_fn(y_true, y_pred, sample_weight=weights)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/metrics.py", line 196, in call replica_local_fn, *args, **kwargs)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py", line 1135, in call_replica_local_fn return fn(*args, **kwargs)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/metrics.py", line 179, in replica_local_fn update_op = self.update_state(*args, **kwargs) # pylint: disable=not-callable

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/metrics_utils.py", line 76, in decorated update_op = update_state_fn(*args, **kwargs)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/keras/metrics.py", line 587, in update_state matches = self._fn(y_true, y_pred, **self._fn_kwargs)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/metrics/keras_metrics.py", line 100, in f1score return fbeta(y_true, y_pred, beta=1)

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/talos/metrics/keras_metrics.py", line 88, in fbeta if K.sum(K.round(K.clip(y_true, 0, 1))) == 0:

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 757, in bool self._disallow_bool_casting()

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 526, in _disallow_bool_casting self._disallow_in_graph_mode("using a tf.Tensor as a Python bool")

File "/home/w/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 515, in _disallow_in_graph_mode " this function with @tf.function.".format(task))

OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

Thanks for the answers so far. Any suggestion on how to resolve this? It is indeed related to the TF 2.0 framework.

autonomio / talos