[BUG] Tests failing with tensorflow 2.12.0

[dstansby]

The release of tensorflow 2.12.0 is causing some of our tests to fail. See e.g. https://github.com/brainglobe/cellfinder-core/actions/runs/4503578523/jobs/7926945458.

I don't have time to investigate now, so I'll pin to tensorflow<2.12.0 in dependencies for now.

[dstansby]

One of the traces is

  _____________________________ test_detection_full ______________________________

  signal_array = dask.array<stack, shape=(30, 510, 667), dtype=uint16, chunksize=(1, 510, 667), chunktype=numpy.ndarray>
  background_array = dask.array<stack, shape=(30, 510, 667), dtype=uint16, chunksize=(1, 510, 667), chunktype=numpy.ndarray>

      @pytest.mark.slow
      def test_detection_full(signal_array, background_array):
  >       cells_test = main(
              signal_array,
              background_array,
              voxel_sizes,
              n_free_cpus=0,
          )

  tests/tests/test_integration/test_detection.py:35: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  .tox/py310/lib/python3.10/site-packages/cellfinder_core/main.py:95: in main
      points = classify.main(
  .tox/py310/lib/python3.10/site-packages/cellfinder_core/classify/classify.py:74: in main
      predictions = model.predict(
  .tox/py310/lib/python3.10/site-packages/keras/utils/traceback_utils.py:70: in error_handler
      raise e.with_traceback(filtered_tb) from None
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

  iterator = <tensorflow.python.data.ops.iterator_ops.OwnedIterator object at 0x7f536c61c550>

      def tf__predict_function(iterator):
          """Runs an evaluation execution with a single step."""
          with ag__.FunctionScope('predict_function', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:
              do_return = False
              retval_ = ag__.UndefinedReturnValue()
              try:
                  do_return = True
  >               retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
  E               TypeError: in user code:
  E               
  E                   File "/home/runner/work/cellfinder-core/cellfinder-core/.tox/py310/lib/python3.10/site-packages/keras/engine/training.py", line 2169, in predict_function  *
  E                       return step_function(self, iterator)
  E                   File "/home/runner/work/cellfinder-core/cellfinder-core/.tox/py310/lib/python3.10/site-packages/keras/engine/training.py", line 2155, in step_function  **
  E                       outputs = model.distribute_strategy.run(run_step, args=(data,))
  E                   File "/home/runner/work/cellfinder-core/cellfinder-core/.tox/py310/lib/python3.10/site-packages/keras/engine/training.py", line 2143, in run_step  **
  E                       outputs = model.predict_step(data)
  E                   File "/home/runner/work/cellfinder-core/cellfinder-core/.tox/py310/lib/python3.10/site-packages/keras/engine/training.py", line 2111, in predict_step
  E                       return self(x, training=False)
  E                   File "/home/runner/work/cellfinder-core/cellfinder-core/.tox/py310/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
  E                       raise e.with_traceback(filtered_tb) from None
  E               
  E                   TypeError: Exception encountered when calling layer 'conv1_bn' (type BatchNormalization).
  E                   
  E                   Failed to convert elements of [1, 1, 1, None, 1] to Tensor. Consider casting elements to a supported type. See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.
  E                   
  E                   Call arguments received by layer 'conv1_bn' (type BatchNormalization):
  E                     • inputs=tf.Tensor(shape=(None, None, None, None, 64), dtype=float32)
  E                     • training=False
  E                     • mask=None

  /tmp/__autograph_generated_filerbzi9zef.py:15: TypeError

[dstansby]

I've opened https://github.com/tensorflow/tensorflow/issues/60102 for this

[dstansby]

I spent about half an hour trying to work towards a self-contianed reproducible example of this, and struggled - I think working this out is going to take quite a bit of time getting used to how the classification code works and interfaces with tensorflow.

[dstansby]

Here's a small example to reproduce the error. I'll edit this if I manage to get it any smaller.

from pathlib import Path

import numpy as np
from imlib.cells.cells import Cell

from cellfinder_core.classify.cube_generator import CubeGeneratorFromFile
from cellfinder_core.classify.tools import get_model

points = [Cell((50, 50, 50), 1)]
signal_array = np.random.rand(100, 100, 100)
background_array = signal_array
voxel_sizes = [5, 2, 2]
network_voxel_sizes = (5, 1, 1)

inference_generator = CubeGeneratorFromFile(
    points,
    signal_array,
    background_array,
    voxel_sizes,
    network_voxel_sizes,
)

model_weights = Path(
    "/Users/dstansby/.cellfinder/model_weights/resnet50_tv.h5"
)
model = get_model(
    existing_model=None,
    model_weights=model_weights,
    network_depth="50-layer",
    inference=True,
)

predictions = model.predict(
    inference_generator,
)

[adamltyson]

Closing now we've moved to torch, this issue no longer exists :tada:

[dstansby]

🥳