Problem in training the unet_all_bands

[mboumahdi]

Hi, I'm training to train the model unet_all_bands to generate all sentinel-2 bands, I generated the TF records with 20m bands I used all the default parameters but I get this error

Call arguments received: • inputs=tf.Tensor(shape=(None, 256, 256, 4), dtype=float32) Traceback (most recent call last): File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception yield File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/distribute/mirrored_run.py", line 342, in run self.main_result = self.main_fn(*self.main_args, *self.main_kwargs) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 689, in wrapper return converted_call(f, args, kwargs, options=options) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 377, in converted_call return _call_unconverted(f, args, kwargs, options) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 458, in _call_unconverted return f(args, **kwargs) File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1783, in run_step outputs = model.predict_step(data) File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1751, in predict_step return self(x, training=False) File "/opt/otbtf/lib/python3/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/opt/otbtf/lib/python3/site-packages/keras/layers/convolutional.py", line 3556, in call raise ValueError('Argument cropping must be ' ValueError: Exception encountered when calling layer "s2_estim_pad256" (type Cropping2D).

Argument cropping must be greater than the input shape. Received: inputs.shape=(None, 256, 256, 4), and cropping=((256, 256), (256, 256))

Call arguments received: • inputs=tf.Tensor(shape=(None, 256, 256, 4), dtype=float32) 2023-08-14 21:42:23 INFO Error reported to Coordinator: Exception encountered when calling layer "s2_estim_pad256" (type Cropping2D).

Argument cropping must be greater than the input shape. Received: inputs.shape=(None, 256, 256, 4), and cropping=((256, 256), (256, 256))

Call arguments received: • inputs=tf.Tensor(shape=(None, 256, 256, 4), dtype=float32) Traceback (most recent call last): File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/training/coordinator.py", line 293, in stop_on_exception yield File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/distribute/mirrored_run.py", line 342, in run self.main_result = self.main_fn(*self.main_args, *self.main_kwargs) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 689, in wrapper return converted_call(f, args, kwargs, options=options) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 377, in converted_call return _call_unconverted(f, args, kwargs, options) File "/opt/otbtf/lib/python3/site-packages/tensorflow/python/autograph/impl/api.py", line 458, in _call_unconverted return f(args, **kwargs) File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1783, in run_step outputs = model.predict_step(data) File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1751, in predict_step return self(x, training=False) File "/opt/otbtf/lib/python3/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/opt/otbtf/lib/python3/site-packages/keras/layers/convolutional.py", line 3556, in call raise ValueError('Argument cropping must be ' ValueError: Exception encountered when calling layer "s2_estim_pad256" (type Cropping2D).

Argument cropping must be greater than the input shape. Received: inputs.shape=(None, 256, 256, 4), and cropping=((256, 256), (256, 256))

Call arguments received: • inputs=tf.Tensor(shape=(None, 256, 256, 4), dtype=float32) Traceback (most recent call last): File "models/train_from_tfrecords.py", line 262, in system.run_and_terminate(main) File "/usr/local/lib/python3.8/dist-packages/decloud/core/system.py", line 199, in run_and_terminate sys.exit(main(args=sys.argv[1:])) File "models/train_from_tfrecords.py", line 244, in main model.fit(tf_ds_train, File "/opt/otbtf/lib/python3/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler raise e.with_traceback(filtered_tb) from None File "/usr/local/lib/python3.8/dist-packages/decloud/core/summary.py", line 71, in on_epoch_end predicted = self.model.predict(self.test_data) ValueError: in user code:

File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1801, in predict_function  *
    return step_function(self, iterator)
File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1790, in step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
    raise value
File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1783, in run_step  **
    outputs = model.predict_step(data)
File "/opt/otbtf/lib/python3/site-packages/keras/engine/training.py", line 1751, in predict_step
    return self(x, training=False)
File "/opt/otbtf/lib/python3/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
File "/opt/otbtf/lib/python3/site-packages/keras/layers/convolutional.py", line 3556, in call
    raise ValueError('Argument `cropping` must be '

ValueError: Exception encountered when calling layer "s2_estim_pad256" (type Cropping2D).

Argument `cropping` must be greater than the input shape. Received: inputs.shape=(None, 256, 256, 4), and cropping=((256, 256), (256, 256))

Call arguments received:
  • inputs=tf.Tensor(shape=(None, 256, 256, 4), dtype=float32)

[remicres]

Hello @meryeme-25

The 20m bands have not been extensively tested for now, but it should work. Did you have succeed with 10m bands only? let me know

In the meantime we will investigate this issue

[mboumahdi]

Hello @remicres I tried also with 10m bands and I had the same problem

[remicres]

We can reproduce the problem with otbtf 4.1.0, it looks like something broke with recent Keras versions.

For now I only can suggest you to build decloud on an older otbtf image (3.2.0 or 3.3.0 should be fine).

We are investigating the problem right now

[remicres]

Hi @meryeme-25

We should have fixed the bug now. You should be able to work with decloud with the latest master branch!