search

deephealthproject / use-case-pipelines

Use case pipelines based on EDDL and ECVL libraries. Different tasks (e.g. classification and segmentation) and datasets (e.g. MNIST, ISIC, and PNEUMOTHORAX) are taken into account.
Other
2 stars 5 forks source link

DataGenerator leaks batches #16

Closed MicheleCancilla closed 3 years ago

MicheleCancilla commented 3 years ago

DataGenerator sometimes skips some batches:

Starting validation:
Validation: Epoch 0/149 (batch 0/24) -  categorical_accuracy: 0.85
Validation: Epoch 0/149 (batch 1/24) -  categorical_accuracy: 0.55
Validation: Epoch 0/149 (batch 2/24) -  categorical_accuracy: 0.325
Validation: Epoch 0/149 (batch 3/24) -  categorical_accuracy: 0.45
Validation: Epoch 0/149 (batch 4/24) -  categorical_accuracy: 0.425
Validation: Epoch 0/149 (batch 5/24) -  categorical_accuracy: 0.425
Validation: Epoch 0/149 (batch 6/24) -  categorical_accuracy: 0.95
Validation: Epoch 0/149 (batch 7/24) -  categorical_accuracy: 0.9
Validation: Epoch 0/149 (batch 8/24) -  categorical_accuracy: 0.9
Validation: Epoch 0/149 (batch 9/24) -  categorical_accuracy: 0.775
Validation: Epoch 0/149 (batch 10/24) -  categorical_accuracy: 0.925
Validation: Epoch 0/149 (batch 11/24) -  categorical_accuracy: 0.875
Validation: Epoch 0/149 (batch 12/24) -  categorical_accuracy: 0.85
Validation: Epoch 0/149 (batch 13/24) -  categorical_accuracy: 0.95
Validation: Epoch 0/149 (batch 14/24) -  categorical_accuracy: 0.95
Validation: Epoch 0/149 (batch 15/24) -  categorical_accuracy: 0.9
Validation: Epoch 0/149 (batch 16/24) -  categorical_accuracy: 0.7
Validation: Epoch 0/149 (batch 17/24) -  categorical_accuracy: 0.65
Validation: Epoch 0/149 (batch 18/24) -  categorical_accuracy: 0.675
Validation: Epoch 0/149 (batch 19/24) -  categorical_accuracy: 0.6
Validation: Epoch 0/149 (batch **20/24**) -  categorical_accuracy: 0.275  <-- 21-24 have been skipped
Validation categorical accuracy: 0.709524

Or it saves them for later:

Starting validation:
Validation: Epoch 1/149 (batch 0/24) -  categorical_accuracy: 0.15
Validation: Epoch 1/149 (batch 1/24) -  categorical_accuracy: 0.25
Validation: Epoch 1/149 (batch 2/24) -  categorical_accuracy: 0.45
Validation: Epoch 1/149 (batch 3/24) -  categorical_accuracy: 0.2
Validation: Epoch 1/149 (batch 4/24) -  categorical_accuracy: 0.875
Validation: Epoch 1/149 (batch 5/24) -  categorical_accuracy: 0.575
Validation: Epoch 1/149 (batch 6/24) -  categorical_accuracy: 0.5
Validation: Epoch 1/149 (batch 7/24) -  categorical_accuracy: 0.575
Validation: Epoch 1/149 (batch 8/24) -  categorical_accuracy: 0.45
Validation: Epoch 1/149 (batch 9/24) -  categorical_accuracy: 0.525
Validation: Epoch 1/149 (batch 10/24) -  categorical_accuracy: 0.975
Validation: Epoch 1/149 (batch 11/24) -  categorical_accuracy: 0.875
Validation: Epoch 1/149 (batch 12/24) -  categorical_accuracy: 0.775
Validation: Epoch 1/149 (batch 13/24) -  categorical_accuracy: 0.875
Validation: Epoch 1/149 (batch 14/24) -  categorical_accuracy: 0.9
Validation: Epoch 1/149 (batch 15/24) -  categorical_accuracy: 0.775
Validation: Epoch 1/149 (batch 16/24) -  categorical_accuracy: 0.925
Validation: Epoch 1/149 (batch 17/24) -  categorical_accuracy: 0.925
Validation: Epoch 1/149 (batch 18/24) -  categorical_accuracy: 0.95
Validation: Epoch 1/149 (batch 19/24) -  categorical_accuracy: 0.875
Validation: Epoch 1/149 (batch 20/24) -  categorical_accuracy: 0.8
Validation: Epoch 1/149 (batch 21/24) -  categorical_accuracy: 0.7
Validation: Epoch 1/149 (batch 22/24) -  categorical_accuracy: 0.425
Validation: Epoch 1/149 (batch 23/24) -  categorical_accuracy: 0.7
Validation: Epoch 1/149 (batch 24/24) -  categorical_accuracy: 0.65
Validation: Epoch 1/149 (batch 25/24) -  categorical_accuracy: 0.15 <--- ???

Tip: use sequential LoadBatch for validation and DataGenerator for training.

lauracanalini commented 3 years ago

DataGenerator was removed with PR #19