Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape. Histogram error?

[EndritPJ]

Hi,

I'm having an issue training a model to segment 5 different classes. The dataset I'm using contains 10 MRI images each with 5 different labels. These labels are consistent for each instance (I checked in the .nii.gz files).

The labels are as follows: 0 1 2 64 71

When I train for 3 classes (0, 1 and 2) the training works fine. When I train for 5, I get the below error.

InvalidArgumentError (see above for traceback): Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape

From looking at past posts, I changed my config and histogram ref files, yet I still cannot train. Is it an issue with the labels not being in a sequential order?

[train-images]
path_to_search = ./training/
filename_contains = training_axial_full_
filename_not_contains = label
spatial_window_size = (144, 144, 144)
interp_order = 1
axcodes=(A, R, S)

[train-labels]
path_to_search = ./training/
filename_contains = training_axial_full_, _label_B
spatial_window_size = (144, 144, 144)
interp_order = 0
axcodes=(A, R, S)

[test-images]
path_to_search = ./testing/
filename_contains = testing_axial_full_
filename_not_contains = label
spatial_window_size = (144, 144, 144)
interp_order = 1
axcodes=(A, R, S)

############################## system configuration sections
[SYSTEM]
cuda_devices = ""
num_threads = 2
num_gpus = 1
model_dir = ./models/
dataset_split_file = ./models/dataset_split.csv
queue_length = 10

[NETWORK]
name = dense_vnet
batch_size = 2
normalisation = True
volume_padding_size = 0
window_sampling = resize
histogram_ref_file = ./models/histogram_ref_file.txt

[TRAINING]
sample_per_volume = 1
lr = 0.001
loss_type = Dice
starting_iter = 0
save_every_n = 200
max_iter = 2000
# data augmentation
rotation_angle = (90, 180, 270)
random_flipping_axes = 1
scaling_percentage = (0.8, 1.2)
tensorboard_every_n = 20

[INFERENCE]
border = (0, 0, 0)
inference_iter = 2000
output_interp_order = 0
spatial_window_size = (144, 144, 144)
validation_every_n = 5
exclude_fraction_for_validation = 0.2
dataset_to_infer = test-images
save_seg_dir = ./testing/
output_postfix = _segmentation_niftynet

############################ custom configuration sections
[SEGMENTATION]
image = train-images
label = train-labels
label_normalisation = False
output_prob = False
num_classes = 5

Running this generated a histogram_ref_file.txt, but then failed at the first iteration of training. I then added 2 lines to the histogram .txt, and it looks like so:

train-images 0.0 0.0 0.0 0.0 0.0 0.2530471005484592 1.924915148926776 7.82734619781791 17.384864102727136 22.401117425949785 28.65465884299126 49.433686447119086 100.0
train-labels_label-from 0 1 2 64 71
train-labels_label-to 0 1 2 3 4

The histogram_ref_file points to the correct location, which should enforce the labels to be 0 1 2 3 4. The num_classes=5. My .nii.gz files all have consistent labeling. Can anyone suggest why the training is failing?

[EndritPJ]

Update: Looks like the training is working now, I manually changed the label numbering in the .nii.gz files to be sequential, though it could also be done with a script. I'm still curious to know as to why it wasn't training before if anyone has any further explanations.