Exception has occurred: OperatorNotAllowedInGraphError

[TongbaoYang]

A OperatorNotAllowedInGraphError throw out when I train AnimeGAN, witch I choose 'wgan-gp' as the defualt training gan. Exception trace log as follows:

['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages', '/hy-tmp/AnimeGANv2-master/tools/..']
init test
WARNING:tensorflow:From main.py:111: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.

WARNING:tensorflow:From main.py:112: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From main.py:112: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2022-03-24 14:53:45.629430: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2022-03-24 14:53:45.660248: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2022-03-24 14:53:45.660960: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c658ac42d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-03-24 14:53:45.660998: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2022-03-24 14:53:45.665129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2022-03-24 14:53:45.934579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55c658b497a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-03-24 14:53:45.934642: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2022-03-24 14:53:45.935606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:8d:00.0
2022-03-24 14:53:45.936059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-03-24 14:53:45.938338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2022-03-24 14:53:45.940088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2022-03-24 14:53:45.940606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2022-03-24 14:53:45.944158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2022-03-24 14:53:45.946792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2022-03-24 14:53:45.951124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2022-03-24 14:53:45.952244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2022-03-24 14:53:45.952294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2022-03-24 14:53:45.957530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-03-24 14:53:45.957555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2022-03-24 14:53:45.957564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2022-03-24 14:53:45.958626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15061 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:8d:00.0, compute capability: 7.0)
WARNING:tensorflow:From /hy-tmp/AnimeGANv2-master/AnimeGANv2.py:55: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

npy file loaded -------  vgg19_weight/vgg19.npy

##### Information #####
# gan type :  wgan-gp
# light :  False
# dataset :  Paprika
# max dataset number :  6656
# batch_size :  20
# epoch :  500
# init_epoch :  0
# training image size [H, W] :  [256, 256]
# g_adv_weight,d_adv_weight,con_weight,sty_weight,color_weight,tv_weight :  300.0 300.0 2.0 0.6 50.0 0.1
# init_lr,g_lr,d_lr :  0.0002 8e-05 0.00016
# training_rate G -- D: 1 : 1

WARNING:tensorflow:From /hy-tmp/AnimeGANv2-master/AnimeGANv2.py:96: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /hy-tmp/AnimeGANv2-master/net/generator.py:41: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

WARNING:tensorflow:From /hy-tmp/AnimeGANv2-master/net/generator.py:58: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /hy-tmp/AnimeGANv2-master/AnimeGANv2.py:122: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.python-2022.2.1924087327/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "main.py", line 131, in <module>
    main()
  File "main.py", line 117, in main
    gan.build_model()
  File "/hy-tmp/AnimeGANv2-master/AnimeGANv2.py", line 156, in build_model
    GP = self.gradient_panalty(real=self.anime, fake=self.generated)
  File "/hy-tmp/AnimeGANv2-master/AnimeGANv2.py", line 125, in gradient_panalty
    logit, _= self.discriminator(interpolated, reuse=True, scope=scope)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 547, in __iter__
    self._disallow_iteration()
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 543, in _disallow_iteration
    self._disallow_in_graph_mode("iterating over `tf.Tensor`")
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 523, in _disallow_in_graph_mode
    " this function with @tf.function.".format(task))
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

I'm not familiar with tensorflow 1.x, does any body can help fix this issus.

[ManluZhang]

Did you use the tensorflow 2 to run the script?

[TongbaoYang]

Did you use the tensorflow 2 to run the script?

I use 1.15.x to run this train.py script. It's fine when run with defuat configure, but throw the error when specific 'wgan-gp' as the running gan.

[ManluZhang]

Sorry, I didn't try the 'wgan-gp'. Can't help.

[TachibanaYoshino]

File "/hy-tmp/AnimeGANv2-master/AnimeGANv2.py", line 125, in gradientpanalty logit, = self.discriminator(interpolated, reuse=True, scope=scope) refer to it to modify.

[d4rkm0nkey]

I have the same problem with dragan, wgan-lp and wgan-gp. gan, lsgan and hinge are working. Does anybody know how to fix this?

Traceback (most recent call last):
  File "train.py", line 103, in <module>
    main()
  File "train.py", line 93, in main
    gan.build_model()
  File "/home/.../Documents/masterthesis/repos/AnimeGANv2/AnimeGANv2.py", line 144, in build_model
    GP = self.gradient_panalty(real=self.anime, fake=self.generated)
  File "/home/.../Documents/masterthesis/repos/AnimeGANv2/AnimeGANv2.py", line 114, in gradient_panalty
    logit, _= self.discriminator(interpolated, reuse=True, scope=scope)
  File "/home/.../anaconda3/envs/AnimeGAN/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 547, in __iter__
    self._disallow_iteration()
  File "/home/.../anaconda3/envs/AnimeGAN/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 543, in _disallow_iteration
    self._disallow_in_graph_mode("iterating over `tf.Tensor`")
  File "/home/.../anaconda3/envs/AnimeGAN/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 523, in _disallow_in_graph_mode
    " this function with @tf.function.".format(task))
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.