search

mittagessen / kraken

OCR engine for all the languages
http://kraken.re
Apache License 2.0
673 stars 125 forks source link

ketos segtrain not picking the best model? #581

Closed johnlockejrr closed 2 months ago

johnlockejrr commented 3 months ago

I'm not sure this is an issue or a missunderstanding, anyway, I finetuned a segmentation model on a small data but kraken picks the first output model although following models seem to have higher accuracy.

ketos segtrain -d cuda:0 -f alto -t output.txt -q early -tl --resize both -i /home/incognito/kraken-train/sam_finetuning/ubma_sam_v2.mlmodel -o /home/incognito/kraken-train/sam_finetuning/ubma_sam_v3/ubma_sam_v3
Trainable params: 1.3 M
Non-trainable params: 0
Total params: 1.3 M
Total estimated model params size (MB): 5
stage 0/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.23it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.476 val_freq_iu: 0.916 early_stopping: 0/10 0.47646
stage 1/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.32it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.486 val_freq_iu: 0.92 early_stopping: 0/10 0.48553
stage 2/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.32it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.493 val_freq_iu: 0.92 early_stopping: 0/10 0.49255
stage 3/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.30it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.498 val_freq_iu: 0.922 early_stopping: 0/10 0.49753
stage 4/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.40it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.479 val_freq_iu: 0.915 early_stopping: 1/10 0.49753
stage 5/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.38it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.489 val_freq_iu: 0.917 early_stopping: 2/10 0.49753
stage 6/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.44it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.498 val_freq_iu: 0.923 early_stopping: 0/10 0.49806
stage 7/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.33it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.492 val_freq_iu: 0.92 early_stopping: 1/10 0.49806
stage 8/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.29it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.507 val_freq_iu: 0.926 early_stopping: 0/10 0.50689
stage 9/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.38it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.489 val_freq_iu: 0.921 early_stopping: 1/10 0.50689
stage 10/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.35it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.5 val_freq_iu: 0.923 early_stopping: 2/10 0.50689
stage 11/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.34it/s val_accuracy: 0.988 val_mean_acc: 0.988 val_mean_iu: 0.471 val_freq_iu: 0.898 early_stopping: 3/10 0.50689
stage 12/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.31it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.51 val_freq_iu: 0.92 early_stopping: 0/10 0.51016
stage 13/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.31it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.512 val_freq_iu: 0.922 early_stopping: 0/10 0.51179
stage 14/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.34it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.502 val_freq_iu: 0.921 early_stopping: 1/10 0.51179
stage 15/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.32it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.52 val_freq_iu: 0.927 early_stopping: 0/10 0.51965
stage 16/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.30it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.521 val_freq_iu: 0.928 early_stopping: 0/10 0.52124
stage 17/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.32it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.503 val_freq_iu: 0.921 early_stopping: 1/10 0.52124
stage 18/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.27it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.518 val_freq_iu: 0.925 early_stopping: 2/10 0.52124
stage 19/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.25it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.514 val_freq_iu: 0.926 early_stopping: 3/10 0.52124
stage 20/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.35it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.506 val_freq_iu: 0.922 early_stopping: 4/10 0.52124
stage 21/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.41it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.492 val_freq_iu: 0.918 early_stopping: 5/10 0.52124
stage 22/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.28it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.517 val_freq_iu: 0.925 early_stopping: 6/10 0.52124
stage 23/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.36it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.495 val_freq_iu: 0.917 early_stopping: 7/10 0.52124
stage 24/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.40it/s val_accuracy: 0.992 val_mean_acc: 0.992 val_mean_iu: 0.522 val_freq_iu: 0.929 early_stopping: 0/10 0.52154
stage 25/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.28it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.503 val_freq_iu: 0.921 early_stopping: 1/10 0.52154
stage 26/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.33it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.505 val_freq_iu: 0.923 early_stopping: 2/10 0.52154
stage 27/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.37it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.493 val_freq_iu: 0.916 early_stopping: 3/10 0.52154
stage 28/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.29it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.499 val_freq_iu: 0.919 early_stopping: 4/10 0.52154
stage 29/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.37it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.504 val_freq_iu: 0.921 early_stopping: 5/10 0.52154
stage 30/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.34it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.495 val_freq_iu: 0.915 early_stopping: 6/10 0.52154
stage 31/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.32it/s val_accuracy: 0.99 val_mean_acc: 0.99 val_mean_iu: 0.486 val_freq_iu: 0.914 early_stopping: 7/10 0.52154
stage 32/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.35it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.496 val_freq_iu: 0.92 early_stopping: 8/10 0.52154
stage 33/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:16 • 0:00:00 3.38it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.51 val_freq_iu: 0.924 early_stopping: 9/10 0.52154
stage 34/∞ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56/56 0:00:17 • 0:00:00 3.28it/s val_accuracy: 0.991 val_mean_acc: 0.991 val_mean_iu: 0.511 val_freq_iu: 0.926 early_stopping: 10/10 0.52154
Moving best model /home/incognito/kraken-train/sam_finetuning/ubma_sam_v3/ubma_sam_v3_0.mlmodel (0.5473847389221191) to /home/incognito/kraken-train/sam_finetuning/ubma_sam_v3/ubma_sam_v3_best.mlmodel
mittagessen commented 2 months ago

Sorry for only getting back to you on this now. The validation metric is val_mean_iu not pixel accuracy which is most likely the reason for it picking the original model (as there wasn't any improvement). But the metrics aren't really representative of segmentation quality so using early stopping for segmentation training is not advised. Rather evaluate the output after a fixed number of epochs (your dataset is rather small so 50-100 is reasonable).

johnlockejrr commented 2 months ago

No problem, better later than never :) Thanks for the tip, I'll consider it!