Pyapetnet incompatible with tensorflow >= 2.8

[sbaete]

Hey Georg,

We have a new server, so I had to reinstall pyapetnet. I followed your instructions and used your environment.yml. However, the code crashed with the error below. It seems it is an incompatibility with newer versions of tensorflow. After downgrading to Tensorflow version 2.7.1, the code worked again.

Best regards,

Steven

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2023-08-17 17:04:13.914716: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-17 17:04:13.931590: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/layers/core/lambda_layer.py:303: UserWarning: tensorflow.python.keras.utils.multi_gpu_utils is not loaded, but a Lambda layer uses it. It may cause errors.
  function = cls._parse_function_from_config(config, custom_objects,
Traceback (most recent call last):
  File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/bin/pyapetnet_predict_from_nifti", line 8, in <module>
    sys.exit(main())
  File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/pyapetnet/predict_from_nifti.py", line 148, in main
    pred[...,i] = model.predict(x).squeeze()
  File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'functional_3/functional_1/batchnorm_ind_0_0/FusedBatchNormV3' defined at (most recent call last):
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/bin/pyapetnet_predict_from_nifti", line 8, in <module>
      sys.exit(main())
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/pyapetnet/predict_from_nifti.py", line 148, in main
      pred[...,i] = model.predict(x).squeeze()
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/training.py", line 1982, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/training.py", line 1801, in predict_function
      return step_function(self, iterator)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/training.py", line 1790, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/training.py", line 1783, in run_step
      outputs = model.predict_step(data)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/training.py", line 1751, in predict_step
      return self(x, training=False)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/functional.py", line 451, in call
      return self._run_internal_graph(
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/functional.py", line 589, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/engine/base_layer.py", line 1096, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/layers/normalization/batch_normalization.py", line 767, in call
      outputs = self._fused_batch_norm(inputs, training=training)
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/layers/normalization/batch_normalization.py", line 623, in _fused_batch_norm
      output, mean, variance = control_flow_util.smart_cond(
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/utils/control_flow_util.py", line 105, in smart_cond
      return tf.__internal__.smart_cond.smart_cond(
    File "/home/baetes01/anaconda3/envs/pyapetnet1.5tris/lib/python3.10/site-packages/keras/layers/normalization/batch_normalization.py", line 605, in _fused_batch_norm_inference
      return tf.compat.v1.nn.fused_batch_norm(
Node: 'functional_3/functional_1/batchnorm_ind_0_0/FusedBatchNormV3'
scale must have the same number of elements as the channels of x, got 15 and 1
     [[{{node functional_3/functional_1/batchnorm_ind_0_0/FusedBatchNormV3}}]] [Op:__inference_predict_function_14204]
_ind_0_0/FusedBatchNormV3}}]] [Op:__inference_predict_function_14204]

[gschramm]

Hi Steven,

thanks for bringing this up. I just relocated back to Leuven, so I don't have GPU server access right now (will change beginning of Sep). I just tried to replicate the problem on my M1 Mac with TF v2.13 using a prediction from nifti. There everything works fine, even with the latest TF version.

To further debug this, two questions:

1. Which OS system are you using? Are you using a CUDA GPU?
2. Can you send me the exact call that triggered the bug? (I used the pyapetnet_predict_from_nifti and the demo data sets)

Bests, Georg

[sbaete]

Hey Georg,

Thank you for looking into this.

To answer your questions, 1) We use Ubuntu 22.04.2 LTS. Unfortunately, we don't have a CUDA GPU installed on this server 2) To avoid any other setup-issues causing problems, I went ahead and created a new anaconda-environment and followed the installation steps in the Readme.md, using the pip install. Then I cloned the repository to obtain the demo data and ran the command: ~/pyapetnetdebug/demo_data$ pyapetnet_predict_from_nifti brainweb_06_osem.nii brainweb_06_t1.nii S2_osem_b10_fdg_pe2i --show

I have attached a log of the whole installation process until I got to the error.

Steven

pyapetnetdebug.txt conda_env_installed_packages.txt

[gschramm]

Hi Steven,

I just tested pyapetnet_predict_from_nifti with pyapetnet v1.5.1 and tensorflow 2.13. on Ubuntu 18.04 without GPU. For me everything works as expected and the output looks correct.

So I am bit puzzled to understand what went wrong on your side. Did you have a look again?

(pyapetnet) gschra2@fermi:~/nge/pyapetnet/demo_data$ conda list | grep pyapetnet
# packages in environment at /users/nexuz/gschra2/.conda/envs/pyapetnet:
pyapetnet                 1.5.1                    pypi_0    pypi

(pyapetnet) gschra2@fermi:~/nge/pyapetnet/demo_data$ conda list | grep tensorflow
tensorflow                2.13.0                   pypi_0    pypi
tensorflow-estimator      2.13.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.34.0                   pypi_0    pypi

pyapetnet) gschra2@fermi:~/nge/pyapetnet/demo_data$ pyapetnet_predict_from_nifti brainweb_06_osem.nii brainweb_06_t1.nii S2_osem_b10_fdg_pe2i
2023-09-14 16:26:30.377466: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-09-14 16:26:30.425998: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-09-14 16:26:30.426604: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-09-14 16:26:32.228709: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
/users/nexuz/gschra2/.conda/envs/pyapetnet/lib/python3.11/site-packages/keras/src/layers/core/lambda_layer.py:327: UserWarning: tensorflow.python.keras.utils.multi_gpu_utils is not loaded, but a Lambda layer uses it. It may cause errors.
  function = cls._parse_function_from_config(
1/1 [==============================] - 8s 8s/step

[gschramm]

should be fixed in v1.5.4