search

BiaPyX / BiaPy

Open source Python library for building bioimage analysis pipelines
https://BiaPyX.github.io
MIT License
123 stars 28 forks source link

Unstable training for lucchi and lucchi_pp datasets #25

Closed HenryLyu closed 1 year ago

HenryLyu commented 1 year ago

Dear authors, thank you very much for your excellent paper and the efforts to make the code available. I am trying to reproduce your result on the lucchi and lucchi_pp datasets, finding that jaccard_index is not improving during training (i.e. it reduced quickly to 0.0000e+00 at epoch 0 and stayed unchanged). However, the training process is fine for the kasthuri_pp dataset. And I have tried both SGD and ADAM optimizers, but the problem persisted. The model architecture is the Attention U-net. What could be the problem with this scenario?

The training log is attached as follows:

#####################

TRAIN THE MODEL

#####################

build callbacks done Epoch 1/360

1/297 [..............................] - ETA: 24:24 - loss: 0.7848 - jaccard_index: 0.1074 2/297 [..............................] - ETA: 26s - loss: 0.6453 - jaccard_index: 0.0728
3/297 [..............................] - ETA: 26s - loss: 1.2558 - jaccard_index: 0.0562 4/297 [..............................] - ETA: 26s - loss: 1.0312 - jaccard_index: 0.0424 5/297 [..............................] - ETA: 26s - loss: 0.9182 - jaccard_index: 0.0357 6/297 [..............................] - ETA: 26s - loss: 0.8367 - jaccard_index: 0.0298 7/297 [..............................] - ETA: 26s - loss: 0.7619 - jaccard_index: 0.0256 8/297 [..............................] - ETA: 26s - loss: 0.6990 - jaccard_index: 0.0224 9/297 [..............................] - ETA: 26s - loss: 0.6426 - jaccard_index: 0.0199 10/297 [>.............................] - ETA: 25s - loss: 0.6153 - jaccard_index: 0.0179 11/297 [>.............................] - ETA: 25s - loss: 0.5751 - jaccard_index: 0.0163 12/297 [>.............................] - ETA: 25s - loss: 0.5460 - jaccard_index: 0.0149 13/297 [>.............................] - ETA: 25s - loss: 0.5177 - jaccard_index: 0.0138 14/297 [>.............................] - ETA: 25s - loss: 0.4983 - jaccard_index: 0.0128 15/297 [>.............................] - ETA: 25s - loss: 0.4713 - jaccard_index: 0.0119 16/297 [>.............................] - ETA: 25s - loss: 0.4528 - jaccard_index: 0.0112 17/297 [>.............................] - ETA: 25s - loss: 0.4358 - jaccard_index: 0.0105 18/297 [>.............................] - ETA: 25s - loss: 0.4341 - jaccard_index: 0.0099 19/297 [>.............................] - ETA: 24s - loss: 0.4269 - jaccard_index: 0.0094 20/297 [=>............................] - ETA: 24s - loss: 0.4184 - jaccard_index: 0.0094 21/297 [=>............................] - ETA: 24s - loss: 0.4093 - jaccard_index: 0.0090 22/297 [=>............................] - ETA: 24s - loss: 0.4009 - jaccard_index: 0.0086 23/297 [=>............................] - ETA: 24s - loss: 0.3968 - jaccard_index: 0.0082 24/297 [=>............................] - ETA: 24s - loss: 0.3923 - jaccard_index: 0.0079 25/297 [=>............................] - ETA: 24s - loss: 0.3873 - jaccard_index: 0.0075 26/297 [=>............................] - ETA: 24s - loss: 0.3802 - jaccard_index: 0.0073 27/297 [=>............................] - ETA: 24s - loss: 0.3717 - jaccard_index: 0.0070 28/297 [=>............................] - ETA: 24s - loss: 0.3627 - jaccard_index: 0.0067 29/297 [=>............................] - ETA: 24s - loss: 0.3624 - jaccard_index: 0.0065 30/297 [==>...........................] - ETA: 24s - loss: 0.3552 - jaccard_index: 0.0063 31/297 [==>...........................] - ETA: 24s - loss: 0.3516 - jaccard_index: 0.0061 32/297 [==>...........................] - ETA: 24s - loss: 0.3504 - jaccard_index: 0.0059 33/297 [==>...........................] - ETA: 24s - loss: 0.3444 - jaccard_index: 0.0057 34/297 [==>...........................] - ETA: 24s - loss: 0.3397 - jaccard_index: 0.0055 35/297 [==>...........................] - ETA: 24s - loss: 0.3401 - jaccard_index: 0.0054 36/297 [==>...........................] - ETA: 24s - loss: 0.3377 - jaccard_index: 0.0052 37/297 [==>...........................] - ETA: 23s - loss: 0.3358 - jaccard_index: 0.0051 38/297 [==>...........................] - ETA: 23s - loss: 0.3375 - jaccard_index: 0.0050 39/297 [==>...........................] - ETA: 23s - loss: 0.3353 - jaccard_index: 0.0048 40/297 [===>..........................] - ETA: 23s - loss: 0.3339 - jaccard_index: 0.0047 41/297 [===>..........................] - ETA: 23s - loss: 0.3311 - jaccard_index: 0.0046 42/297 [===>..........................] - ETA: 23s - loss: 0.3271 - jaccard_index: 0.0045 43/297 [===>..........................] - ETA: 23s - loss: 0.3235 - jaccard_index: 0.0044 44/297 [===>..........................] - ETA: 23s - loss: 0.3217 - jaccard_index: 0.0043 45/297 [===>..........................] - ETA: 23s - loss: 0.3197 - jaccard_index: 0.0042 46/297 [===>..........................] - ETA: 23s - loss: 0.3181 - jaccard_index: 0.0041 47/297 [===>..........................] - ETA: 23s - loss: 0.3162 - jaccard_index: 0.0040 48/297 [===>..........................] - ETA: 22s - loss: 0.3156 - jaccard_index: 0.0039 49/297 [===>..........................] - ETA: 22s - loss: 0.3144 - jaccard_index: 0.0038 50/297 [====>.........................] - ETA: 22s - loss: 0.3118 - jaccard_index: 0.0038 51/297 [====>.........................] - ETA: 22s - loss: 0.3106 - jaccard_index: 0.0037 52/297 [====>.........................] - ETA: 22s - loss: 0.3119 - jaccard_index: 0.0036 53/297 [====>.........................] - ETA: 22s - loss: 0.3093 - jaccard_index: 0.0036 54/297 [====>.........................] - ETA: 22s - loss: 0.3096 - jaccard_index: 0.0035 55/297 [====>.........................] - ETA: 22s - loss: 0.3115 - jaccard_index: 0.0034 56/297 [====>.........................] - ETA: 22s - loss: 0.3104 - jaccard_index: 0.0034 57/297 [====>.........................] - ETA: 22s - loss: 0.3109 - jaccard_index: 0.0033 58/297 [====>.........................] - ETA: 22s - loss: 0.3093 - jaccard_index: 0.0033 59/297 [====>.........................] - ETA: 21s - loss: 0.3086 - jaccard_index: 0.0032 60/297 [=====>........................] - ETA: 21s - loss: 0.3078 - jaccard_index: 0.0031 61/297 [=====>........................] - ETA: 21s - loss: 0.3073 - jaccard_index: 0.0031 62/297 [=====>........................] - ETA: 21s - loss: 0.3077 - jaccard_index: 0.0030 63/297 [=====>........................] - ETA: 21s - loss: 0.3072 - jaccard_index: 0.0030 64/297 [=====>........................] - ETA: 21s - loss: 0.3052 - jaccard_index: 0.0029 65/297 [=====>........................] - ETA: 21s - loss: 0.3047 - jaccard_index: 0.0029 66/297 [=====>........................] - ETA: 21s - loss: 0.3039 - jaccard_index: 0.0029 67/297 [=====>........................] - ETA: 21s - loss: 0.3036 - jaccard_index: 0.0028 68/297 [=====>........................] - ETA: 21s - loss: 0.3026 - jaccard_index: 0.0028 69/297 [=====>........................] - ETA: 20s - loss: 0.3021 - jaccard_index: 0.0027 70/297 [======>.......................] - ETA: 20s - loss: 0.3023 - jaccard_index: 0.0027 71/297 [======>.......................] - ETA: 20s - loss: 0.3023 - jaccard_index: 0.0027 72/297 [======>.......................] - ETA: 20s - loss: 0.3010 - jaccard_index: 0.0026 73/297 [======>.......................] - ETA: 20s - loss: 0.3000 - jaccard_index: 0.0026 74/297 [======>.......................] - ETA: 20s - loss: 0.2989 - jaccard_index: 0.0025 75/297 [======>.......................] - ETA: 20s - loss: 0.2972 - jaccard_index: 0.0025 76/297 [======>.......................] - ETA: 20s - loss: 0.2959 - jaccard_index: 0.0025 77/297 [======>.......................] - ETA: 20s - loss: 0.2938 - jaccard_index: 0.0024 78/297 [======>.......................] - ETA: 20s - loss: 0.2909 - jaccard_index: 0.0024 79/297 [======>.......................] - ETA: 20s - loss: 0.2903 - jaccard_index: 0.0024 80/297 [=======>......................] - ETA: 19s - loss: 0.2880 - jaccard_index: 0.0024 81/297 [=======>......................] - ETA: 19s - loss: 0.2880 - jaccard_index: 0.0023 82/297 [=======>......................] - ETA: 19s - loss: 0.2874 - jaccard_index: 0.0023 83/297 [=======>......................] - ETA: 19s - loss: 0.2860 - jaccard_index: 0.0023 84/297 [=======>......................] - ETA: 19s - loss: 0.2865 - jaccard_index: 0.0022 85/297 [=======>......................] - ETA: 19s - loss: 0.2872 - jaccard_index: 0.0022 86/297 [=======>......................] - ETA: 19s - loss: 0.2869 - jaccard_index: 0.0022 87/297 [=======>......................] - ETA: 19s - loss: 0.2883 - jaccard_index: 0.0022 88/297 [=======>......................] - ETA: 19s - loss: 0.2886 - jaccard_index: 0.0023 89/297 [=======>......................] - ETA: 19s - loss: 0.2887 - jaccard_index: 0.0023 90/297 [========>.....................] - ETA: 18s - loss: 0.2881 - jaccard_index: 0.0023 91/297 [========>.....................] - ETA: 18s - loss: 0.2878 - jaccard_index: 0.0022 92/297 [========>.....................] - ETA: 18s - loss: 0.2870 - jaccard_index: 0.0022 93/297 [========>.....................] - ETA: 18s - loss: 0.2864 - jaccard_index: 0.0022 94/297 [========>.....................] - ETA: 18s - loss: 0.2856 - jaccard_index: 0.0022 95/297 [========>.....................] - ETA: 18s - loss: 0.2857 - jaccard_index: 0.0021 96/297 [========>.....................] - ETA: 18s - loss: 0.2847 - jaccard_index: 0.0021 97/297 [========>.....................] - ETA: 18s - loss: 0.2837 - jaccard_index: 0.0021 98/297 [========>.....................] - ETA: 18s - loss: 0.2829 - jaccard_index: 0.0021 99/297 [=========>....................] - ETA: 18s - loss: 0.2825 - jaccard_index: 0.0021 100/297 [=========>....................] - ETA: 18s - loss: 0.2814 - jaccard_index: 0.0020 101/297 [=========>....................] - ETA: 17s - loss: 0.2810 - jaccard_index: 0.0020 102/297 [=========>....................] - ETA: 17s - loss: 0.2817 - jaccard_index: 0.0020 103/297 [=========>....................] - ETA: 17s - loss: 0.2834 - jaccard_index: 0.0020 104/297 [=========>....................] - ETA: 17s - loss: 0.2831 - jaccard_index: 0.0020 105/297 [=========>....................] - ETA: 17s - loss: 0.2829 - jaccard_index: 0.0019 106/297 [=========>....................] - ETA: 17s - loss: 0.2822 - jaccard_index: 0.0019 107/297 [=========>....................] - ETA: 17s - loss: 0.2824 - jaccard_index: 0.0019 108/297 [=========>....................] - ETA: 17s - loss: 0.2823 - jaccard_index: 0.0019 109/297 [==========>...................] - ETA: 17s - loss: 0.2831 - jaccard_index: 0.0019 110/297 [==========>...................] - ETA: 17s - loss: 0.2824 - jaccard_index: 0.0019 111/297 [==========>...................] - ETA: 17s - loss: 0.2821 - jaccard_index: 0.0019 112/297 [==========>...................] - ETA: 16s - loss: 0.2825 - jaccard_index: 0.0019 113/297 [==========>...................] - ETA: 16s - loss: 0.2823 - jaccard_index: 0.0018 114/297 [==========>...................] - ETA: 16s - loss: 0.2809 - jaccard_index: 0.0018 115/297 [==========>...................] - ETA: 16s - loss: 0.2802 - jaccard_index: 0.0018 116/297 [==========>...................] - ETA: 16s - loss: 0.2801 - jaccard_index: 0.0018 117/297 [==========>...................] - ETA: 16s - loss: 0.2806 - jaccard_index: 0.0018 118/297 [==========>...................] - ETA: 16s - loss: 0.2806 - jaccard_index: 0.0018 119/297 [===========>..................] - ETA: 16s - loss: 0.2801 - jaccard_index: 0.0018 120/297 [===========>..................] - ETA: 16s - loss: 0.2795 - jaccard_index: 0.0017 121/297 [===========>..................] - ETA: 16s - loss: 0.2804 - jaccard_index: 0.0017 122/297 [===========>..................] - ETA: 16s - loss: 0.2795 - jaccard_index: 0.0017 123/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0017 124/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0017 125/297 [===========>..................] - ETA: 15s - loss: 0.2793 - jaccard_index: 0.0017 126/297 [===========>..................] - ETA: 15s - loss: 0.2796 - jaccard_index: 0.0017 127/297 [===========>..................] - ETA: 15s - loss: 0.2790 - jaccard_index: 0.0016 128/297 [===========>..................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0016 129/297 [============>.................] - ETA: 15s - loss: 0.2794 - jaccard_index: 0.0016 130/297 [============>.................] - ETA: 15s - loss: 0.2792 - jaccard_index: 0.0016 131/297 [============>.................] - ETA: 15s - loss: 0.2792 - jaccard_index: 0.0016 132/297 [============>.................] - ETA: 15s - loss: 0.2782 - jaccard_index: 0.0016 133/297 [============>.................] - ETA: 15s - loss: 0.2779 - jaccard_index: 0.0016 134/297 [============>.................] - ETA: 14s - loss: 0.2778 - jaccard_index: 0.0016 135/297 [============>.................] - ETA: 14s - loss: 0.2782 - jaccard_index: 0.0015 136/297 [============>.................] - ETA: 14s - loss: 0.2780 - jaccard_index: 0.0015 137/297 [============>.................] - ETA: 14s - loss: 0.2775 - jaccard_index: 0.0015 138/297 [============>.................] - ETA: 14s - loss: 0.2771 - jaccard_index: 0.0015 139/297 [=============>................] - ETA: 14s - loss: 0.2769 - jaccard_index: 0.0015 140/297 [=============>................] - ETA: 14s - loss: 0.2764 - jaccard_index: 0.0015 141/297 [=============>................] - ETA: 14s - loss: 0.2766 - jaccard_index: 0.0015 142/297 [=============>................] - ETA: 14s - loss: 0.2773 - jaccard_index: 0.0015 143/297 [=============>................] - ETA: 14s - loss: 0.2773 - jaccard_index: 0.0015 144/297 [=============>................] - ETA: 14s - loss: 0.2765 - jaccard_index: 0.0014 145/297 [=============>................] - ETA: 13s - loss: 0.2760 - jaccard_index: 0.0014 146/297 [=============>................] - ETA: 13s - loss: 0.2760 - jaccard_index: 0.0014 147/297 [=============>................] - ETA: 13s - loss: 0.2756 - jaccard_index: 0.0014 148/297 [=============>................] - ETA: 13s - loss: 0.2749 - jaccard_index: 0.0014 149/297 [==============>...............] - ETA: 13s - loss: 0.2739 - jaccard_index: 0.0014 150/297 [==============>...............] - ETA: 13s - loss: 0.2733 - jaccard_index: 0.0014 151/297 [==============>...............] - ETA: 13s - loss: 0.2748 - jaccard_index: 0.0014 152/297 [==============>...............] - ETA: 13s - loss: 0.2743 - jaccard_index: 0.0014 153/297 [==============>...............] - ETA: 13s - loss: 0.2738 - jaccard_index: 0.0014 154/297 [==============>...............] - ETA: 13s - loss: 0.2737 - jaccard_index: 0.0014 155/297 [==============>...............] - ETA: 12s - loss: 0.2740 - jaccard_index: 0.0013 156/297 [==============>...............] - ETA: 12s - loss: 0.2734 - jaccard_index: 0.0013 157/297 [==============>...............] - ETA: 12s - loss: 0.2733 - jaccard_index: 0.0013 158/297 [==============>...............] - ETA: 12s - loss: 0.2734 - jaccard_index: 0.0013 159/297 [===============>..............] - ETA: 12s - loss: 0.2730 - jaccard_index: 0.0013 160/297 [===============>..............] - ETA: 12s - loss: 0.2727 - jaccard_index: 0.0013 161/297 [===============>..............] - ETA: 12s - loss: 0.2726 - jaccard_index: 0.0013 162/297 [===============>..............] - ETA: 12s - loss: 0.2720 - jaccard_index: 0.0013 163/297 [===============>..............] - ETA: 12s - loss: 0.2721 - jaccard_index: 0.0013 164/297 [===============>..............] - ETA: 12s - loss: 0.2713 - jaccard_index: 0.0013 165/297 [===============>..............] - ETA: 12s - loss: 0.2709 - jaccard_index: 0.0013 166/297 [===============>..............] - ETA: 11s - loss: 0.2702 - jaccard_index: 0.0013 167/297 [===============>..............] - ETA: 11s - loss: 0.2694 - jaccard_index: 0.0012 168/297 [===============>..............] - ETA: 11s - loss: 0.2684 - jaccard_index: 0.0012 169/297 [================>.............] - ETA: 11s - loss: 0.2680 - jaccard_index: 0.0012 170/297 [================>.............] - ETA: 11s - loss: 0.2687 - jaccard_index: 0.0012 171/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012 172/297 [================>.............] - ETA: 11s - loss: 0.2684 - jaccard_index: 0.0012 173/297 [================>.............] - ETA: 11s - loss: 0.2677 - jaccard_index: 0.0012 174/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012 175/297 [================>.............] - ETA: 11s - loss: 0.2685 - jaccard_index: 0.0012 176/297 [================>.............] - ETA: 11s - loss: 0.2686 - jaccard_index: 0.0012 177/297 [================>.............] - ETA: 10s - loss: 0.2680 - jaccard_index: 0.0012 178/297 [================>.............] - ETA: 10s - loss: 0.2675 - jaccard_index: 0.0012 179/297 [=================>............] - ETA: 10s - loss: 0.2670 - jaccard_index: 0.0012 180/297 [=================>............] - ETA: 10s - loss: 0.2671 - jaccard_index: 0.0012 181/297 [=================>............] - ETA: 10s - loss: 0.2667 - jaccard_index: 0.0012 182/297 [=================>............] - ETA: 10s - loss: 0.2664 - jaccard_index: 0.0011 183/297 [=================>............] - ETA: 10s - loss: 0.2662 - jaccard_index: 0.0011 184/297 [=================>............] - ETA: 10s - loss: 0.2656 - jaccard_index: 0.0011 185/297 [=================>............] - ETA: 10s - loss: 0.2652 - jaccard_index: 0.0011 186/297 [=================>............] - ETA: 10s - loss: 0.2657 - jaccard_index: 0.0011 187/297 [=================>............] - ETA: 10s - loss: 0.2651 - jaccard_index: 0.0011 188/297 [=================>............] - ETA: 9s - loss: 0.2648 - jaccard_index: 0.0011 189/297 [==================>...........] - ETA: 9s - loss: 0.2649 - jaccard_index: 0.0011 190/297 [==================>...........] - ETA: 9s - loss: 0.2656 - jaccard_index: 0.0011 191/297 [==================>...........] - ETA: 9s - loss: 0.2662 - jaccard_index: 0.0011 192/297 [==================>...........] - ETA: 9s - loss: 0.2661 - jaccard_index: 0.0011 193/297 [==================>...........] - ETA: 9s - loss: 0.2657 - jaccard_index: 0.0011 194/297 [==================>...........] - ETA: 9s - loss: 0.2657 - jaccard_index: 0.0011 195/297 [==================>...........] - ETA: 9s - loss: 0.2656 - jaccard_index: 0.0011 196/297 [==================>...........] - ETA: 9s - loss: 0.2653 - jaccard_index: 0.0011 197/297 [==================>...........] - ETA: 9s - loss: 0.2648 - jaccard_index: 0.0011 198/297 [===================>..........] - ETA: 9s - loss: 0.2647 - jaccard_index: 0.0011 199/297 [===================>..........] - ETA: 8s - loss: 0.2642 - jaccard_index: 0.0010 200/297 [===================>..........] - ETA: 8s - loss: 0.2642 - jaccard_index: 0.0010 201/297 [===================>..........] - ETA: 8s - loss: 0.2643 - jaccard_index: 0.0010 202/297 [===================>..........] - ETA: 8s - loss: 0.2638 - jaccard_index: 0.0010 203/297 [===================>..........] - ETA: 8s - loss: 0.2645 - jaccard_index: 0.0010 204/297 [===================>..........] - ETA: 8s - loss: 0.2650 - jaccard_index: 0.0010 205/297 [===================>..........] - ETA: 8s - loss: 0.2644 - jaccard_index: 0.0010 206/297 [===================>..........] - ETA: 8s - loss: 0.2646 - jaccard_index: 0.0010 207/297 [===================>..........] - ETA: 8s - loss: 0.2644 - jaccard_index: 0.0010 208/297 [====================>.........] - ETA: 8s - loss: 0.2649 - jaccard_index: 0.0010 209/297 [====================>.........] - ETA: 8s - loss: 0.2655 - jaccard_index: 9.9654e-04 210/297 [====================>.........] - ETA: 7s - loss: 0.2652 - jaccard_index: 9.9180e-04 211/297 [====================>.........] - ETA: 7s - loss: 0.2652 - jaccard_index: 9.8710e-04 212/297 [====================>.........] - ETA: 7s - loss: 0.2651 - jaccard_index: 9.8244e-04 213/297 [====================>.........] - ETA: 7s - loss: 0.2650 - jaccard_index: 9.7783e-04 214/297 [====================>.........] - ETA: 7s - loss: 0.2651 - jaccard_index: 9.7326e-04 215/297 [====================>.........] - ETA: 7s - loss: 0.2649 - jaccard_index: 9.6873e-04 216/297 [====================>.........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.6425e-04 217/297 [====================>.........] - ETA: 7s - loss: 0.2646 - jaccard_index: 9.5980e-04 218/297 [=====================>........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.5540e-04 219/297 [=====================>........] - ETA: 7s - loss: 0.2644 - jaccard_index: 9.5104e-04 220/297 [=====================>........] - ETA: 7s - loss: 0.2639 - jaccard_index: 9.4671e-04 221/297 [=====================>........] - ETA: 6s - loss: 0.2633 - jaccard_index: 9.4243e-04 222/297 [=====================>........] - ETA: 6s - loss: 0.2634 - jaccard_index: 9.3819e-04 223/297 [=====================>........] - ETA: 6s - loss: 0.2630 - jaccard_index: 9.3398e-04 224/297 [=====================>........] - ETA: 6s - loss: 0.2633 - jaccard_index: 9.2981e-04 225/297 [=====================>........] - ETA: 6s - loss: 0.2630 - jaccard_index: 9.2568e-04 226/297 [=====================>........] - ETA: 6s - loss: 0.2625 - jaccard_index: 9.2158e-04 227/297 [=====================>........] - ETA: 6s - loss: 0.2623 - jaccard_index: 9.1752e-04 228/297 [======================>.......] - ETA: 6s - loss: 0.2623 - jaccard_index: 9.1350e-04 229/297 [======================>.......] - ETA: 6s - loss: 0.2618 - jaccard_index: 9.0951e-04 230/297 [======================>.......] - ETA: 6s - loss: 0.2620 - jaccard_index: 9.0555e-04 231/297 [======================>.......] - ETA: 6s - loss: 0.2615 - jaccard_index: 9.0163e-04 232/297 [======================>.......] - ETA: 5s - loss: 0.2613 - jaccard_index: 8.9775e-04 233/297 [======================>.......] - ETA: 5s - loss: 0.2611 - jaccard_index: 8.9389e-04 234/297 [======================>.......] - ETA: 5s - loss: 0.2605 - jaccard_index: 8.9007e-04 235/297 [======================>.......] - ETA: 5s - loss: 0.2600 - jaccard_index: 8.8629e-04 236/297 [======================>.......] - ETA: 5s - loss: 0.2595 - jaccard_index: 8.8253e-04 237/297 [======================>.......] - ETA: 5s - loss: 0.2593 - jaccard_index: 8.7881e-04 238/297 [=======================>......] - ETA: 5s - loss: 0.2592 - jaccard_index: 8.7511e-04 239/297 [=======================>......] - ETA: 5s - loss: 0.2594 - jaccard_index: 8.7145e-04 240/297 [=======================>......] - ETA: 5s - loss: 0.2592 - jaccard_index: 8.6782e-04 241/297 [=======================>......] - ETA: 5s - loss: 0.2589 - jaccard_index: 8.6422e-04 242/297 [=======================>......] - ETA: 5s - loss: 0.2585 - jaccard_index: 8.6065e-04 243/297 [=======================>......] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.5711e-04 244/297 [=======================>......] - ETA: 4s - loss: 0.2581 - jaccard_index: 8.5359e-04 245/297 [=======================>......] - ETA: 4s - loss: 0.2576 - jaccard_index: 8.5011e-04 246/297 [=======================>......] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.4666e-04 247/297 [=======================>......] - ETA: 4s - loss: 0.2582 - jaccard_index: 8.4323e-04 248/297 [========================>.....] - ETA: 4s - loss: 0.2580 - jaccard_index: 8.3983e-04 249/297 [========================>.....] - ETA: 4s - loss: 0.2581 - jaccard_index: 8.3645e-04 250/297 [========================>.....] - ETA: 4s - loss: 0.2576 - jaccard_index: 8.3311e-04 251/297 [========================>.....] - ETA: 4s - loss: 0.2571 - jaccard_index: 8.2979e-04 252/297 [========================>.....] - ETA: 4s - loss: 0.2574 - jaccard_index: 8.2650e-04 253/297 [========================>.....] - ETA: 4s - loss: 0.2569 - jaccard_index: 8.2323e-04 254/297 [========================>.....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.1999e-04 255/297 [========================>.....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.1677e-04 256/297 [========================>.....] - ETA: 3s - loss: 0.2563 - jaccard_index: 8.1358e-04 257/297 [========================>.....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.1042e-04 258/297 [=========================>....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.0728e-04 259/297 [=========================>....] - ETA: 3s - loss: 0.2566 - jaccard_index: 8.0416e-04 260/297 [=========================>....] - ETA: 3s - loss: 0.2568 - jaccard_index: 8.0107e-04 261/297 [=========================>....] - ETA: 3s - loss: 0.2569 - jaccard_index: 7.9800e-04 262/297 [=========================>....] - ETA: 3s - loss: 0.2575 - jaccard_index: 7.9495e-04 263/297 [=========================>....] - ETA: 3s - loss: 0.2576 - jaccard_index: 7.9193e-04 264/297 [=========================>....] - ETA: 3s - loss: 0.2575 - jaccard_index: 7.8893e-04 265/297 [=========================>....] - ETA: 2s - loss: 0.2572 - jaccard_index: 7.8595e-04 266/297 [=========================>....] - ETA: 2s - loss: 0.2575 - jaccard_index: 7.8300e-04 267/297 [=========================>....] - ETA: 2s - loss: 0.2573 - jaccard_index: 7.8006e-04 268/297 [==========================>...] - ETA: 2s - loss: 0.2571 - jaccard_index: 7.7715e-04 269/297 [==========================>...] - ETA: 2s - loss: 0.2568 - jaccard_index: 7.7426e-04 270/297 [==========================>...] - ETA: 2s - loss: 0.2568 - jaccard_index: 7.7140e-04 271/297 [==========================>...] - ETA: 2s - loss: 0.2563 - jaccard_index: 7.6855e-04 272/297 [==========================>...] - ETA: 2s - loss: 0.2560 - jaccard_index: 7.6572e-04 273/297 [==========================>...] - ETA: 2s - loss: 0.2556 - jaccard_index: 7.6292e-04 274/297 [==========================>...] - ETA: 2s - loss: 0.2560 - jaccard_index: 7.6014e-04 275/297 [==========================>...] - ETA: 2s - loss: 0.2561 - jaccard_index: 7.5737e-04 276/297 [==========================>...] - ETA: 1s - loss: 0.2559 - jaccard_index: 7.5463e-04 277/297 [==========================>...] - ETA: 1s - loss: 0.2558 - jaccard_index: 7.5190e-04 278/297 [===========================>..] - ETA: 1s - loss: 0.2556 - jaccard_index: 7.4920e-04 279/297 [===========================>..] - ETA: 1s - loss: 0.2557 - jaccard_index: 7.4651e-04 280/297 [===========================>..] - ETA: 1s - loss: 0.2558 - jaccard_index: 7.4385e-04 281/297 [===========================>..] - ETA: 1s - loss: 0.2559 - jaccard_index: 7.4271e-04 282/297 [===========================>..] - ETA: 1s - loss: 0.2556 - jaccard_index: 7.4008e-04 283/297 [===========================>..] - ETA: 1s - loss: 0.2552 - jaccard_index: 7.3746e-04 284/297 [===========================>..] - ETA: 1s - loss: 0.2546 - jaccard_index: 7.3487e-04 285/297 [===========================>..] - ETA: 1s - loss: 0.2550 - jaccard_index: 7.3229e-04 286/297 [===========================>..] - ETA: 1s - loss: 0.2546 - jaccard_index: 7.2973e-04 287/297 [===========================>..] - ETA: 0s - loss: 0.2546 - jaccard_index: 7.2718e-04 288/297 [============================>.] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.2466e-04 289/297 [============================>.] - ETA: 0s - loss: 0.2544 - jaccard_index: 7.2215e-04 290/297 [============================>.] - ETA: 0s - loss: 0.2541 - jaccard_index: 7.1966e-04 291/297 [============================>.] - ETA: 0s - loss: 0.2545 - jaccard_index: 7.1719e-04 292/297 [============================>.] - ETA: 0s - loss: 0.2544 - jaccard_index: 7.1473e-04 293/297 [============================>.] - ETA: 0s - loss: 0.2545 - jaccard_index: 7.1229e-04 294/297 [============================>.] - ETA: 0s - loss: 0.2549 - jaccard_index: 7.0987e-04 295/297 [============================>.] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.0746e-04 296/297 [============================>.] - ETA: 0s - loss: 0.2548 - jaccard_index: 7.0507e-04 297/297 [==============================] - ETA: 0s - loss: 0.2547 - jaccard_index: 7.0270e-04

Epoch 00001: val_loss improved from inf to 0.25942, saving model to content\output\reproduce_lucchi_pp_10\checkpoints\model_weights_reproduce_lucchi_pp_10_10.h5

297/297 [==============================] - 34s 98ms/step - loss: 0.2547 - jaccard_index: 7.0270e-04 - val_loss: 0.2594 - val_jaccard_index: 0.0000e+00 Epoch 2/360

1/297 [..............................] - ETA: 39s - loss: 0.3499 - jaccard_index: 0.0000e+00 2/297 [..............................] - ETA: 26s - loss: 0.2899 - jaccard_index: 0.0000e+00 3/297 [..............................] - ETA: 26s - loss: 0.2773 - jaccard_index: 0.0000e+00 4/297 [..............................] - ETA: 26s - loss: 0.2892 - jaccard_index: 0.0000e+00 5/297 [..............................] - ETA: 26s - loss: 0.2869 - jaccard_index: 0.0000e+00 6/297 [..............................] - ETA: 25s - loss: 0.3050 - jaccard_index: 0.0000e+00 7/297 [..............................] - ETA: 25s - loss: 0.2988 - jaccard_index: 0.0000e+00 8/297 [..............................] - ETA: 25s - loss: 0.2908 - jaccard_index: 0.0000e+00 9/297 [..............................] - ETA: 26s - loss: 0.2805 - jaccard_index: 0.0000e+00 10/297 [>.............................] - ETA: 26s - loss: 0.2854 - jaccard_index: 0.0000e+00 11/297 [>.............................] - ETA: 25s - loss: 0.2756 - jaccard_index: 0.0000e+00 12/297 [>.............................] - ETA: 25s - loss: 0.2721 - jaccard_index: 0.0000e+00 13/297 [>.............................] - ETA: 25s - loss: 0.2647 - jaccard_index: 0.0000e+00 14/297 [>.............................] - ETA: 26s - loss: 0.2620 - jaccard_index: 0.0000e+00 15/297 [>.............................] - ETA: 26s - loss: 0.2517 - jaccard_index: 0.0000e+00 16/297 [>.............................] - ETA: 26s - loss: 0.2469 - jaccard_index: 0.0000e+00 17/297 [>.............................] - ETA: 26s - loss: 0.2415 - jaccard_index: 0.0000e+00 18/297 [>.............................] - ETA: 25s - loss: 0.2484 - jaccard_index: 0.0000e+00 19/297 [>.............................] - ETA: 25s - loss: 0.2506 - jaccard_index: 0.0000e+00 20/297 [=>............................] - ETA: 25s - loss: 0.2493 - jaccard_index: 0.0000e+00 21/297 [=>............................] - ETA: 25s - loss: 0.2488 - jaccard_index: 0.0000e+00 22/297 [=>............................] - ETA: 25s - loss: 0.2465 - jaccard_index: 0.0000e+00 23/297 [=>............................] - ETA: 25s - loss: 0.2478 - jaccard_index: 0.0000e+00 24/297 [=>............................] - ETA: 25s - loss: 0.2494 - jaccard_index: 0.0000e+00 25/297 [=>............................] - ETA: 25s - loss: 0.2502 - jaccard_index: 0.0000e+00 26/297 [=>............................] - ETA: 25s - loss: 0.2479 - jaccard_index: 0.0000e+00 27/297 [=>............................] - ETA: 25s - loss: 0.2434 - jaccard_index: 0.0000e+00 28/297 [=>............................] - ETA: 25s - loss: 0.2394 - jaccard_index: 0.0000e+00 29/297 [=>............................] - ETA: 25s - loss: 0.2439 - jaccard_index: 0.0000e+00 30/297 [==>...........................] - ETA: 24s - loss: 0.2408 - jaccard_index: 0.0000e+00 31/297 [==>...........................] - ETA: 24s - loss: 0.2397 - jaccard_index: 0.0000e+00 32/297 [==>...........................] - ETA: 24s - loss: 0.2411 - jaccard_index: 0.0000e+00 33/297 [==>...........................] - ETA: 24s - loss: 0.2387 - jaccard_index: 0.0000e+00 34/297 [==>...........................] - ETA: 24s - loss: 0.2369 - jaccard_index: 0.0000e+00 35/297 [==>...........................] - ETA: 24s - loss: 0.2398 - jaccard_index: 0.0000e+00 36/297 [==>...........................] - ETA: 24s - loss: 0.2397 - jaccard_index: 0.0000e+00 37/297 [==>...........................] - ETA: 24s - loss: 0.2408 - jaccard_index: 0.0000e+00 38/297 [==>...........................] - ETA: 24s - loss: 0.2446 - jaccard_index: 0.0000e+00 39/297 [==>...........................] - ETA: 23s - loss: 0.2446 - jaccard_index: 0.0000e+00 40/297 [===>..........................] - ETA: 23s - loss: 0.2452 - jaccard_index: 0.0000e+00 41/297 [===>..........................] - ETA: 23s - loss: 0.2443 - jaccard_index: 0.0000e+00 42/297 [===>..........................] - ETA: 23s - loss: 0.2417 - jaccard_index: 0.0000e+00 43/297 [===>..........................] - ETA: 23s - loss: 0.2399 - jaccard_index: 0.0000e+00 44/297 [===>..........................] - ETA: 23s - loss: 0.2399 - jaccard_index: 0.0000e+00 45/297 [===>..........................] - ETA: 23s - loss: 0.2394 - jaccard_index: 0.0000e+00 46/297 [===>..........................] - ETA: 23s - loss: 0.2394 - jaccard_index: 0.0000e+00 47/297 [===>..........................] - ETA: 23s - loss: 0.2389 - jaccard_index: 0.0000e+00 48/297 [===>..........................] - ETA: 23s - loss: 0.2398 - jaccard_index: 0.0000e+00 49/297 [===>..........................] - ETA: 22s - loss: 0.2398 - jaccard_index: 0.0000e+00 50/297 [====>.........................] - ETA: 22s - loss: 0.2387 - jaccard_index: 0.0000e+00 51/297 [====>.........................] - ETA: 22s - loss: 0.2390 - jaccard_index: 0.0000e+00 52/297 [====>.........................] - ETA: 22s - loss: 0.2411 - jaccard_index: 0.0000e+00 53/297 [====>.........................] - ETA: 22s - loss: 0.2398 - jaccard_index: 0.0000e+00 54/297 [====>.........................] - ETA: 22s - loss: 0.2410 - jaccard_index: 0.0000e+00 55/297 [====>.........................] - ETA: 22s - loss: 0.2440 - jaccard_index: 0.0000e+00 56/297 [====>.........................] - ETA: 22s - loss: 0.2441 - jaccard_index: 0.0000e+00 57/297 [====>.........................] - ETA: 22s - loss: 0.2454 - jaccard_index: 0.0000e+00 58/297 [====>.........................] - ETA: 22s - loss: 0.2447 - jaccard_index: 0.0000e+00 59/297 [====>.........................] - ETA: 22s - loss: 0.2455 - jaccard_index: 0.0000e+00 60/297 [=====>........................] - ETA: 21s - loss: 0.2459 - jaccard_index: 0.0000e+00 61/297 [=====>........................] - ETA: 21s - loss: 0.2464 - jaccard_index: 0.0000e+00 62/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00 63/297 [=====>........................] - ETA: 21s - loss: 0.2481 - jaccard_index: 0.0000e+00 64/297 [=====>........................] - ETA: 21s - loss: 0.2472 - jaccard_index: 0.0000e+00 65/297 [=====>........................] - ETA: 21s - loss: 0.2475 - jaccard_index: 0.0000e+00 66/297 [=====>........................] - ETA: 21s - loss: 0.2475 - jaccard_index: 0.0000e+00 67/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00 68/297 [=====>........................] - ETA: 21s - loss: 0.2477 - jaccard_index: 0.0000e+00 69/297 [=====>........................] - ETA: 21s - loss: 0.2480 - jaccard_index: 0.0000e+00 70/297 [======>.......................] - ETA: 20s - loss: 0.2488 - jaccard_index: 0.0000e+00 71/297 [======>.......................] - ETA: 20s - loss: 0.2493 - jaccard_index: 0.0000e+00 72/297 [======>.......................] - ETA: 20s - loss: 0.2485 - jaccard_index: 0.0000e+00 73/297 [======>.......................] - ETA: 20s - loss: 0.2481 - jaccard_index: 0.0000e+00 74/297 [======>.......................] - ETA: 20s - loss: 0.2474 - jaccard_index: 0.0000e+00 75/297 [======>.......................] - ETA: 20s - loss: 0.2465 - jaccard_index: 0.0000e+00 76/297 [======>.......................] - ETA: 20s - loss: 0.2459 - jaccard_index: 0.0000e+00 77/297 [======>.......................] - ETA: 20s - loss: 0.2444 - jaccard_index: 0.0000e+00 78/297 [======>.......................] - ETA: 20s - loss: 0.2421 - jaccard_index: 0.0000e+00 79/297 [======>.......................] - ETA: 20s - loss: 0.2420 - jaccard_index: 0.0000e+00 80/297 [=======>......................] - ETA: 19s - loss: 0.2403 - jaccard_index: 0.0000e+00 81/297 [=======>......................] - ETA: 19s - loss: 0.2408 - jaccard_index: 0.0000e+00 82/297 [=======>......................] - ETA: 19s - loss: 0.2409 - jaccard_index: 0.0000e+00 83/297 [=======>......................] - ETA: 19s - loss: 0.2401 - jaccard_index: 0.0000e+00 84/297 [=======>......................] - ETA: 19s - loss: 0.2411 - jaccard_index: 0.0000e+00 85/297 [=======>......................] - ETA: 19s - loss: 0.2421 - jaccard_index: 0.0000e+00 86/297 [=======>......................] - ETA: 19s - loss: 0.2427 - jaccard_index: 0.0000e+00 87/297 [=======>......................] - ETA: 19s - loss: 0.2449 - jaccard_index: 0.0000e+00 88/297 [=======>......................] - ETA: 19s - loss: 0.2453 - jaccard_index: 0.0000e+00 89/297 [=======>......................] - ETA: 19s - loss: 0.2456 - jaccard_index: 0.0000e+00 90/297 [========>.....................] - ETA: 19s - loss: 0.2453 - jaccard_index: 0.0000e+00 91/297 [========>.....................] - ETA: 18s - loss: 0.2455 - jaccard_index: 0.0000e+00 92/297 [========>.....................] - ETA: 18s - loss: 0.2452 - jaccard_index: 0.0000e+00 93/297 [========>.....................] - ETA: 18s - loss: 0.2452 - jaccard_index: 0.0000e+00 94/297 [========>.....................] - ETA: 18s - loss: 0.2447 - jaccard_index: 0.0000e+00 95/297 [========>.....................] - ETA: 18s - loss: 0.2453 - jaccard_index: 0.0000e+00 96/297 [========>.....................] - ETA: 18s - loss: 0.2448 - jaccard_index: 0.0000e+00 97/297 [========>.....................] - ETA: 18s - loss: 0.2442 - jaccard_index: 0.0000e+00 98/297 [========>.....................] - ETA: 18s - loss: 0.2435 - jaccard_index: 0.0000e+00 99/297 [=========>....................] - ETA: 18s - loss: 0.2436 - jaccard_index: 0.0000e+00 100/297 [=========>....................] - ETA: 18s - loss: 0.2430 - jaccard_index: 0.0000e+00 101/297 [=========>....................] - ETA: 18s - loss: 0.2427 - jaccard_index: 0.0000e+00 102/297 [=========>....................] - ETA: 17s - loss: 0.2437 - jaccard_index: 0.0000e+00 103/297 [=========>....................] - ETA: 17s - loss: 0.2458 - jaccard_index: 0.0000e+00 104/297 [=========>....................] - ETA: 17s - loss: 0.2458 - jaccard_index: 0.0000e+00 105/297 [=========>....................] - ETA: 17s - loss: 0.2459 - jaccard_index: 0.0000e+00 106/297 [=========>....................] - ETA: 17s - loss: 0.2456 - jaccard_index: 0.0000e+00 107/297 [=========>....................] - ETA: 17s - loss: 0.2461 - jaccard_index: 0.0000e+00 108/297 [=========>....................] - ETA: 17s - loss: 0.2464 - jaccard_index: 0.0000e+00 109/297 [==========>...................] - ETA: 17s - loss: 0.2474 - jaccard_index: 0.0000e+00 110/297 [==========>...................] - ETA: 17s - loss: 0.2467 - jaccard_index: 0.0000e+00 111/297 [==========>...................] - ETA: 17s - loss: 0.2467 - jaccard_index: 0.0000e+00 112/297 [==========>...................] - ETA: 16s - loss: 0.2474 - jaccard_index: 0.0000e+00 113/297 [==========>...................] - ETA: 16s - loss: 0.2474 - jaccard_index: 0.0000e+00 114/297 [==========>...................] - ETA: 16s - loss: 0.2464 - jaccard_index: 0.0000e+00 115/297 [==========>...................] - ETA: 16s - loss: 0.2459 - jaccard_index: 0.0000e+00 116/297 [==========>...................] - ETA: 16s - loss: 0.2460 - jaccard_index: 0.0000e+00 117/297 [==========>...................] - ETA: 16s - loss: 0.2466 - jaccard_index: 0.0000e+00 118/297 [==========>...................] - ETA: 16s - loss: 0.2469 - jaccard_index: 0.0000e+00 119/297 [===========>..................] - ETA: 16s - loss: 0.2467 - jaccard_index: 0.0000e+00 120/297 [===========>..................] - ETA: 16s - loss: 0.2465 - jaccard_index: 0.0000e+00 121/297 [===========>..................] - ETA: 16s - loss: 0.2478 - jaccard_index: 0.0000e+00 122/297 [===========>..................] - ETA: 16s - loss: 0.2471 - jaccard_index: 0.0000e+00 123/297 [===========>..................] - ETA: 15s - loss: 0.2473 - jaccard_index: 0.0000e+00 124/297 [===========>..................] - ETA: 15s - loss: 0.2476 - jaccard_index: 0.0000e+00 125/297 [===========>..................] - ETA: 15s - loss: 0.2477 - jaccard_index: 0.0000e+00 126/297 [===========>..................] - ETA: 15s - loss: 0.2481 - jaccard_index: 0.0000e+00 127/297 [===========>..................] - ETA: 15s - loss: 0.2477 - jaccard_index: 0.0000e+00 128/297 [===========>..................] - ETA: 15s - loss: 0.2482 - jaccard_index: 0.0000e+00 129/297 [============>.................] - ETA: 15s - loss: 0.2485 - jaccard_index: 0.0000e+00 130/297 [============>.................] - ETA: 15s - loss: 0.2486 - jaccard_index: 0.0000e+00 131/297 [============>.................] - ETA: 15s - loss: 0.2489 - jaccard_index: 0.0000e+00 132/297 [============>.................] - ETA: 15s - loss: 0.2482 - jaccard_index: 0.0000e+00 133/297 [============>.................] - ETA: 15s - loss: 0.2480 - jaccard_index: 0.0000e+00 134/297 [============>.................] - ETA: 14s - loss: 0.2482 - jaccard_index: 0.0000e+00 135/297 [============>.................] - ETA: 14s - loss: 0.2488 - jaccard_index: 0.0000e+00 136/297 [============>.................] - ETA: 14s - loss: 0.2488 - jaccard_index: 0.0000e+00 137/297 [============>.................] - ETA: 14s - loss: 0.2486 - jaccard_index: 0.0000e+00 138/297 [============>.................] - ETA: 14s - loss: 0.2484 - jaccard_index: 0.0000e+00 139/297 [=============>................] - ETA: 14s - loss: 0.2482 - jaccard_index: 0.0000e+00 140/297 [=============>................] - ETA: 14s - loss: 0.2479 - jaccard_index: 0.0000e+00 141/297 [=============>................] - ETA: 14s - loss: 0.2483 - jaccard_index: 0.0000e+00 142/297 [=============>................] - ETA: 14s - loss: 0.2493 - jaccard_index: 0.0000e+00 143/297 [=============>................] - ETA: 14s - loss: 0.2495 - jaccard_index: 0.0000e+00 144/297 [=============>................] - ETA: 14s - loss: 0.2489 - jaccard_index: 0.0000e+00 145/297 [=============>................] - ETA: 13s - loss: 0.2484 - jaccard_index: 0.0000e+00 146/297 [=============>................] - ETA: 13s - loss: 0.2487 - jaccard_index: 0.0000e+00 147/297 [=============>................] - ETA: 13s - loss: 0.2484 - jaccard_index: 0.0000e+00 148/297 [=============>................] - ETA: 13s - loss: 0.2478 - jaccard_index: 0.0000e+00 149/297 [==============>...............] - ETA: 13s - loss: 0.2469 - jaccard_index: 0.0000e+00 150/297 [==============>...............] - ETA: 13s - loss: 0.2464 - jaccard_index: 0.0000e+00 151/297 [==============>...............] - ETA: 13s - loss: 0.2480 - jaccard_index: 0.0000e+00 152/297 [==============>...............] - ETA: 13s - loss: 0.2475 - jaccard_index: 0.0000e+00 153/297 [==============>...............] - ETA: 13s - loss: 0.2473 - jaccard_index: 0.0000e+00 154/297 [==============>...............] - ETA: 13s - loss: 0.2473 - jaccard_index: 0.0000e+00 155/297 [==============>...............] - ETA: 13s - loss: 0.2477 - jaccard_index: 0.0000e+00 156/297 [==============>...............] - ETA: 12s - loss: 0.2473 - jaccard_index: 0.0000e+00 157/297 [==============>...............] - ETA: 12s - loss: 0.2474 - jaccard_index: 0.0000e+00 158/297 [==============>...............] - ETA: 12s - loss: 0.2476 - jaccard_index: 0.0000e+00 159/297 [===============>..............] - ETA: 12s - loss: 0.2474 - jaccard_index: 0.0000e+00 160/297 [===============>..............] - ETA: 12s - loss: 0.2471 - jaccard_index: 0.0000e+00 161/297 [===============>..............] - ETA: 12s - loss: 0.2472 - jaccard_index: 0.0000e+00 162/297 [===============>..............] - ETA: 12s - loss: 0.2466 - jaccard_index: 0.0000e+00 163/297 [===============>..............] - ETA: 12s - loss: 0.2469 - jaccard_index: 0.0000e+00 164/297 [===============>..............] - ETA: 12s - loss: 0.2461 - jaccard_index: 0.0000e+00 165/297 [===============>..............] - ETA: 12s - loss: 0.2459 - jaccard_index: 0.0000e+00 166/297 [===============>..............] - ETA: 12s - loss: 0.2454 - jaccard_index: 0.0000e+00 167/297 [===============>..............] - ETA: 11s - loss: 0.2446 - jaccard_index: 0.0000e+00 168/297 [===============>..............] - ETA: 11s - loss: 0.2438 - jaccard_index: 0.0000e+00 169/297 [================>.............] - ETA: 11s - loss: 0.2435 - jaccard_index: 0.0000e+00 170/297 [================>.............] - ETA: 11s - loss: 0.2443 - jaccard_index: 0.0000e+00 171/297 [================>.............] - ETA: 11s - loss: 0.2442 - jaccard_index: 0.0000e+00 172/297 [================>.............] - ETA: 11s - loss: 0.2441 - jaccard_index: 0.0000e+00 173/297 [================>.............] - ETA: 11s - loss: 0.2437 - jaccard_index: 0.0000e+00 174/297 [================>.............] - ETA: 11s - loss: 0.2446 - jaccard_index: 0.0000e+00 175/297 [================>.............] - ETA: 11s - loss: 0.2448 - jaccard_index: 0.0000e+00 176/297 [================>.............] - ETA: 11s - loss: 0.2451 - jaccard_index: 0.0000e+00 177/297 [================>.............] - ETA: 11s - loss: 0.2447 - jaccard_index: 0.0000e+00 178/297 [================>.............] - ETA: 10s - loss: 0.2444 - jaccard_index: 0.0000e+00 179/297 [=================>............] - ETA: 10s - loss: 0.2439 - jaccard_index: 0.0000e+00 180/297 [=================>............] - ETA: 10s - loss: 0.2440 - jaccard_index: 0.0000e+00 181/297 [=================>............] - ETA: 10s - loss: 0.2436 - jaccard_index: 0.0000e+00 182/297 [=================>............] - ETA: 10s - loss: 0.2434 - jaccard_index: 0.0000e+00 183/297 [=================>............] - ETA: 10s - loss: 0.2432 - jaccard_index: 0.0000e+00 184/297 [=================>............] - ETA: 10s - loss: 0.2427 - jaccard_index: 0.0000e+00 185/297 [=================>............] - ETA: 10s - loss: 0.2425 - jaccard_index: 0.0000e+00 186/297 [=================>............] - ETA: 10s - loss: 0.2431 - jaccard_index: 0.0000e+00 187/297 [=================>............] - ETA: 10s - loss: 0.2426 - jaccard_index: 0.0000e+00 188/297 [=================>............] - ETA: 10s - loss: 0.2423 - jaccard_index: 0.0000e+00 189/297 [==================>...........] - ETA: 9s - loss: 0.2425 - jaccard_index: 0.0000e+00 190/297 [==================>...........] - ETA: 9s - loss: 0.2431 - jaccard_index: 0.0000e+00 191/297 [==================>...........] - ETA: 9s - loss: 0.2437 - jaccard_index: 0.0000e+00 192/297 [==================>...........] - ETA: 9s - loss: 0.2438 - jaccard_index: 0.0000e+00 193/297 [==================>...........] - ETA: 9s - loss: 0.2434 - jaccard_index: 0.0000e+00 194/297 [==================>...........] - ETA: 9s - loss: 0.2436 - jaccard_index: 0.0000e+00 195/297 [==================>...........] - ETA: 9s - loss: 0.2436 - jaccard_index: 0.0000e+00 196/297 [==================>...........] - ETA: 9s - loss: 0.2434 - jaccard_index: 0.0000e+00 197/297 [==================>...........] - ETA: 9s - loss: 0.2430 - jaccard_index: 0.0000e+00 198/297 [===================>..........] - ETA: 9s - loss: 0.2430 - jaccard_index: 0.0000e+00 199/297 [===================>..........] - ETA: 8s - loss: 0.2424 - jaccard_index: 0.0000e+00 200/297 [===================>..........] - ETA: 8s - loss: 0.2425 - jaccard_index: 0.0000e+00 201/297 [===================>..........] - ETA: 8s - loss: 0.2427 - jaccard_index: 0.0000e+00 202/297 [===================>..........] - ETA: 8s - loss: 0.2423 - jaccard_index: 0.0000e+00 203/297 [===================>..........] - ETA: 8s - loss: 0.2431 - jaccard_index: 0.0000e+00 204/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00 205/297 [===================>..........] - ETA: 8s - loss: 0.2433 - jaccard_index: 0.0000e+00 206/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00 207/297 [===================>..........] - ETA: 8s - loss: 0.2436 - jaccard_index: 0.0000e+00 208/297 [====================>.........] - ETA: 8s - loss: 0.2441 - jaccard_index: 0.0000e+00 209/297 [====================>.........] - ETA: 8s - loss: 0.2448 - jaccard_index: 0.0000e+00 210/297 [====================>.........] - ETA: 7s - loss: 0.2447 - jaccard_index: 0.0000e+00 211/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00 212/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00 213/297 [====================>.........] - ETA: 7s - loss: 0.2449 - jaccard_index: 0.0000e+00 214/297 [====================>.........] - ETA: 7s - loss: 0.2451 - jaccard_index: 0.0000e+00 215/297 [====================>.........] - ETA: 7s - loss: 0.2450 - jaccard_index: 0.0000e+00 216/297 [====================>.........] - ETA: 7s - loss: 0.2444 - jaccard_index: 0.0000e+00 217/297 [====================>.........] - ETA: 7s - loss: 0.2448 - jaccard_index: 0.0000e+00 218/297 [=====================>........] - ETA: 7s - loss: 0.2447 - jaccard_index: 0.0000e+00 219/297 [=====================>........] - ETA: 7s - loss: 0.2448 - jaccard_index: 0.0000e+00 220/297 [=====================>........] - ETA: 7s - loss: 0.2445 - jaccard_index: 0.0000e+00 221/297 [=====================>........] - ETA: 6s - loss: 0.2439 - jaccard_index: 0.0000e+00 222/297 [=====================>........] - ETA: 6s - loss: 0.2442 - jaccard_index: 0.0000e+00 223/297 [=====================>........] - ETA: 6s - loss: 0.2437 - jaccard_index: 0.0000e+00 224/297 [=====================>........] - ETA: 6s - loss: 0.2441 - jaccard_index: 0.0000e+00 225/297 [=====================>........] - ETA: 6s - loss: 0.2438 - jaccard_index: 0.0000e+00 226/297 [=====================>........] - ETA: 6s - loss: 0.2435 - jaccard_index: 0.0000e+00 227/297 [=====================>........] - ETA: 6s - loss: 0.2434 - jaccard_index: 0.0000e+00 228/297 [======================>.......] - ETA: 6s - loss: 0.2434 - jaccard_index: 0.0000e+00 229/297 [======================>.......] - ETA: 6s - loss: 0.2430 - jaccard_index: 0.0000e+00 230/297 [======================>.......] - ETA: 6s - loss: 0.2432 - jaccard_index: 0.0000e+00 231/297 [======================>.......] - ETA: 6s - loss: 0.2429 - jaccard_index: 0.0000e+00 232/297 [======================>.......] - ETA: 5s - loss: 0.2428 - jaccard_index: 0.0000e+00 233/297 [======================>.......] - ETA: 5s - loss: 0.2427 - jaccard_index: 0.0000e+00 234/297 [======================>.......] - ETA: 5s - loss: 0.2421 - jaccard_index: 0.0000e+00 235/297 [======================>.......] - ETA: 5s - loss: 0.2416 - jaccard_index: 0.0000e+00 236/297 [======================>.......] - ETA: 5s - loss: 0.2413 - jaccard_index: 0.0000e+00 237/297 [======================>.......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00 238/297 [=======================>......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00 239/297 [=======================>......] - ETA: 5s - loss: 0.2414 - jaccard_index: 0.0000e+00 240/297 [=======================>......] - ETA: 5s - loss: 0.2412 - jaccard_index: 0.0000e+00 241/297 [=======================>......] - ETA: 5s - loss: 0.2410 - jaccard_index: 0.0000e+00 242/297 [=======================>......] - ETA: 5s - loss: 0.2407 - jaccard_index: 0.0000e+00 243/297 [=======================>......] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00 244/297 [=======================>......] - ETA: 4s - loss: 0.2405 - jaccard_index: 0.0000e+00 245/297 [=======================>......] - ETA: 4s - loss: 0.2401 - jaccard_index: 0.0000e+00 246/297 [=======================>......] - ETA: 4s - loss: 0.2405 - jaccard_index: 0.0000e+00 247/297 [=======================>......] - ETA: 4s - loss: 0.2406 - jaccard_index: 0.0000e+00 248/297 [========================>.....] - ETA: 4s - loss: 0.2406 - jaccard_index: 0.0000e+00 249/297 [========================>.....] - ETA: 4s - loss: 0.2408 - jaccard_index: 0.0000e+00 250/297 [========================>.....] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00 251/297 [========================>.....] - ETA: 4s - loss: 0.2398 - jaccard_index: 0.0000e+00 252/297 [========================>.....] - ETA: 4s - loss: 0.2403 - jaccard_index: 0.0000e+00 253/297 [========================>.....] - ETA: 4s - loss: 0.2397 - jaccard_index: 0.0000e+00 254/297 [========================>.....] - ETA: 3s - loss: 0.2395 - jaccard_index: 0.0000e+00 255/297 [========================>.....] - ETA: 3s - loss: 0.2397 - jaccard_index: 0.0000e+00 256/297 [========================>.....] - ETA: 3s - loss: 0.2393 - jaccard_index: 0.0000e+00 257/297 [========================>.....] - ETA: 3s - loss: 0.2398 - jaccard_index: 0.0000e+00 258/297 [=========================>....] - ETA: 3s - loss: 0.2397 - jaccard_index: 0.0000e+00 259/297 [=========================>....] - ETA: 3s - loss: 0.2398 - jaccard_index: 0.0000e+00 260/297 [=========================>....] - ETA: 3s - loss: 0.2402 - jaccard_index: 0.0000e+00 261/297 [=========================>....] - ETA: 3s - loss: 0.2404 - jaccard_index: 0.0000e+00 262/297 [=========================>....] - ETA: 3s - loss: 0.2411 - jaccard_index: 0.0000e+00 263/297 [=========================>....] - ETA: 3s - loss: 0.2412 - jaccard_index: 0.0000e+00 264/297 [=========================>....] - ETA: 3s - loss: 0.2412 - jaccard_index: 0.0000e+00 265/297 [=========================>....] - ETA: 2s - loss: 0.2409 - jaccard_index: 0.0000e+00 266/297 [=========================>....] - ETA: 2s - loss: 0.2413 - jaccard_index: 0.0000e+00 267/297 [=========================>....] - ETA: 2s - loss: 0.2411 - jaccard_index: 0.0000e+00 268/297 [==========================>...] - ETA: 2s - loss: 0.2410 - jaccard_index: 0.0000e+00 269/297 [==========================>...] - ETA: 2s - loss: 0.2408 - jaccard_index: 0.0000e+00 270/297 [==========================>...] - ETA: 2s - loss: 0.2407 - jaccard_index: 0.0000e+00 271/297 [==========================>...] - ETA: 2s - loss: 0.2403 - jaccard_index: 0.0000e+00 272/297 [==========================>...] - ETA: 2s - loss: 0.2401 - jaccard_index: 0.0000e+00 273/297 [==========================>...] - ETA: 2s - loss: 0.2398 - jaccard_index: 0.0000e+00 274/297 [==========================>...] - ETA: 2s - loss: 0.2402 - jaccard_index: 0.0000e+00 275/297 [==========================>...] - ETA: 2s - loss: 0.2403 - jaccard_index: 0.0000e+00 276/297 [==========================>...] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00 277/297 [==========================>...] - ETA: 1s - loss: 0.2400 - jaccard_index: 0.0000e+00 278/297 [===========================>..] - ETA: 1s - loss: 0.2399 - jaccard_index: 0.0000e+00 279/297 [===========================>..] - ETA: 1s - loss: 0.2400 - jaccard_index: 0.0000e+00 280/297 [===========================>..] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00 281/297 [===========================>..] - ETA: 1s - loss: 0.2403 - jaccard_index: 0.0000e+00 282/297 [===========================>..] - ETA: 1s - loss: 0.2401 - jaccard_index: 0.0000e+00 283/297 [===========================>..] - ETA: 1s - loss: 0.2397 - jaccard_index: 0.0000e+00 284/297 [===========================>..] - ETA: 1s - loss: 0.2392 - jaccard_index: 0.0000e+00 285/297 [===========================>..] - ETA: 1s - loss: 0.2397 - jaccard_index: 0.0000e+00 286/297 [===========================>..] - ETA: 1s - loss: 0.2394 - jaccard_index: 0.0000e+00 287/297 [===========================>..] - ETA: 0s - loss: 0.2393 - jaccard_index: 0.0000e+00 288/297 [============================>.] - ETA: 0s - loss: 0.2395 - jaccard_index: 0.0000e+00 289/297 [============================>.] - ETA: 0s - loss: 0.2392 - jaccard_index: 0.0000e+00 290/297 [============================>.] - ETA: 0s - loss: 0.2390 - jaccard_index: 0.0000e+00 291/297 [============================>.] - ETA: 0s - loss: 0.2394 - jaccard_index: 0.0000e+00 292/297 [============================>.] - ETA: 0s - loss: 0.2394 - jaccard_index: 0.0000e+00 293/297 [============================>.] - ETA: 0s - loss: 0.2395 - jaccard_index: 0.0000e+00 294/297 [============================>.] - ETA: 0s - loss: 0.2400 - jaccard_index: 0.0000e+00 295/297 [============================>.] - ETA: 0s - loss: 0.2399 - jaccard_index: 0.0000e+00 296/297 [============================>.] - ETA: 0s - loss: 0.2400 - jaccard_index: 0.0000e+00 297/297 [==============================] - ETA: 0s - loss: 0.2399 - jaccard_index: 0.0000e+00

Epoch 00002: val_loss improved from 0.25942 to 0.25855, saving model to content\output\reproduce_lucchi_pp_10\checkpoints\model_weights_reproduce_lucchi_pp_10_10.h5

297/297 [==============================] - 29s 96ms/step - loss: 0.2399 - jaccard_index: 0.0000e+00 - val_loss: 0.2585 - val_jaccard_index: 0.0000e+00

danifranco commented 1 year ago

Hello,

How many epochs did you run? Can you attach here the YAML config file used and the commit you are using? As we are under continuous development of BiaPy.

Thanks,

Dani

HenryLyu commented 1 year ago

Thanks for your response Dani, the jaccard_index starts to grow after training about 150 epochs. Here are the metrics at the end of epoch 420: loss: 0.0824 - jaccard_index: 0.5985 - val_loss: 0.1672 - val_jaccard_index: 0.4564.

Below is the YAML file. Regards

AUGMENTOR: ENABLE: true HFLIP: true RANDOM_ROT: true SHUFFLE_TRAIN_DATA_EACH_EPOCH: true SHUFFLE_VAL_DATA_EACH_EPOCH: true VFLIP: true DATA: CHECK_GENERATORS: false PATCH_SIZE: (256, 256, 1) TEST: IN_MEMORY: true LOAD_GT: true MASK_PATH: content\data\lucchi_pp\test\y PADDING: (32, 32) PATH: content\data\lucchi_pp\test\x TRAIN: IN_MEMORY: true MASK_PATH: content\data\lucchi_pp\train\y PATH: content\data\lucchi_pp\train\x VAL: FROM_TRAIN: true SPLIT_TRAIN: 0.1 MODEL: ARCHITECTURE: attention_unet DROPOUT_VALUES:

danifranco commented 1 year ago

Hello,

We need to check each workflow's notebook but I have just run the 2D Semantic Segmentation one with its default options, i.e. with Attention U-Net, and it's learning smoothly from the first epochs. Will update a minor things of the notebook but you can use this one in the meanwhile:

https://drive.google.com/file/d/1elOzvPxTFBDg-iEiCISvRyquqbaISxus/view?usp=sharing

HenryLyu commented 1 year ago

Thank you for sharing the notebook, Dani! I think it is my data preprocessing issue, the notebook works fine with the Lucchi dataset but not the Lucchi++ dataset. Regards