tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 1 (size = 1) and num_split 4

[lokeycookie]

Hello,

I am able to run the pretrain code for rotation_3d and have its saved weights. In the config file for pretraining, I used "top_architecture": "big_fully". Currently, I am trying to run the finetuning code for rotation_3d. I have a BRATS dataset which are all in numpy files. The BRATS images have dimensiom of (128,128,128,4) and they are already normalised, thus intensity of voxels range from 0 to 1. I am now trying to use BRATS labels of dimension (128,128,128) and they are not in one-hot encoding. Thus, the BRATS labels numpy array contain values of either 0,1,2 or 3. I also did the following changes.

1. In line 107 of this file https://github.com/HealthML/self-supervised-3d-tasks/blob/master/self_supervised_3d_tasks/data/segmentation_task_loader.py, I changed the patch size to (32,32,32) and patches_per_scan to be 64.
2. In line 181 of this file https://github.com/HealthML/self-supervised-3d-tasks/blob/master/self_supervised_3d_tasks/data/segmentation_task_loader.py, I changes n_classes to be 4 as initially I recieved a ValueError.

However, after attempting to run the finetune code, I recieved the following error.

Traceback (most recent call last): File "finetune.py", line 4, in main() File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py", line 418, in main init(run_complex_test, "test") File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/utils/model_utils.py", line 67, in init f(args) File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py", line 354, in run_complex_test lambda: run_single_test(algorithm_def, gen_train, gen_val, True, False, x_test, y_test, lr, File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py", line 247, in try_until_no_nan return func() File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py", line 357, in logging_b_path, kwargs, clipnorm=clipnorm, clipvalue=clipvalue)) File "/home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py", line 174, in run_single_test callbacks=w_callbacks, File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit use_multiprocessing=use_multiprocessing) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit total_epochs=epochs) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 128, in run_one_epoch batch_outs = execution_function(iterator) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 98, in execution_function distributed_function(input_fn)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 568, in call result = self._call(*args, *kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 632, in _call return self._stateless_fn(args, kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2363, in call return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1611, in _filtered_call self.captured_inputs) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1692, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 545, in call ctx=ctx) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from tensorflow.python.framework.errors_impl.InvalidArgumentError: Number of ways to split should evenly divide the split dimension, but got split_dim 1 (size = 1) and num_split 4 [[node model_5/model_4/up_sampling3d/split (defined at /home/svu/e0310071/self-supervised-3d-tasks/self_supervised_3d_tasks/finetune.py:174) ]] [Op:__inference_distributed_function_11489]

Function call stack: distributed_function

Also, I am using TensorFlow version 2.1 and python version 3.6. Are these versions correct?

What does the error mean? Does anyone know how to solve this error?

[lokeycookie]

Hello ,

So recently, I tried with the labels not in one-hot, meaning the labels dimension are (128,128,128). Afterwards, I make the following changes.

1. In line 170 of https://github.com/HealthML/self-supervised-3d-tasks/blob/master/self_supervised_3d_tasks/data/segmentation_task_loader.py, I changed the line to this: origin_row = np.random.randint(0, 1, self.patches_per_scan) origin_col = np.random.randint(0, 1, self.patches_per_scan) origin_dep = np.random.randint(0, 1, self.patches_per_scan)

The patch_size is still at (128,128,128) and patches_per_scan is 3.

2. In line 187 of https://github.com/HealthML/self-supervised-3d-tasks/blob/master/self_supervised_3d_tasks/data/segmentation_task_loader.py, I changed the line to this:

n_classes = 4

I hardcode the number of classes to be 4 since my BRATS dataset labels all have 4 classes. After the following changes, I am able to run finish the finetuning code for all 5 self-supervised algorithms. I also did not have any errors while running the finetuning code and the model is able to train well. I am not exactly sure why the above following changes work for my case. Nonetheless, thank you all for your help!

[aihamtaleb]

Hi @lokeycookie Thanks for letting me know. I also can't tell why the changes you did work, without looking at the data. Good luck! and don't forget to cite our work in your future projects :)