search

yzhangcs / parser

:rocket: State-of-the-art parsers for natural language.
https://parser.yzhang.site/
MIT License
825 stars 138 forks source link

Inconsistent behaviour when using the same file for dev and test #93

Closed MinionAttack closed 2 years ago

MinionAttack commented 2 years ago

Hi, I am training some models and have noticed some strange behaviour. If I specify the DEV file for the validation and test phase, I generally get the same score when training a model, but sometimes (randomly?) the score doesn't match.

Below I show an example of a training with albert-base-v2 the differences are minimal but in some (rare) cases with other models I have seen them a bit bigger. For this part of the traning log file, the affected epochs are: 4, 5, 7, 9, 12, 15, 17, 18, 20, 22, 23, 24, 25, 26, 27, 28, 30, 33, 35, 39, 40, 43, 45, 46, 49, 50 ,51, 54, 56, 57, 58 and 59.

2022-02-04 09:56:46 INFO Epoch 1 / 200:
2022-02-04 09:56:56 INFO lr: 2.5000e-06 - loss: 0.1021 - UP:  0.00% UR:  0.00% UF:  0.00% P:  0.00% R:  0.00% F:  0.00%                                
2022-02-04 09:56:57 INFO dev:  loss: nan - UP:  0.00% UR:  0.00% UF:  0.00% P:  0.00% R:  0.00% F:  0.00%
2022-02-04 09:56:59 INFO test: loss: nan - UP:  0.00% UR:  0.00% UF:  0.00% P:  0.00% R:  0.00% F:  0.00%
2022-02-04 09:56:59 INFO 0:00:12.789335s elapsed

2022-02-04 09:56:59 INFO Epoch 2 / 200:
2022-02-04 09:57:09 INFO lr: 5.0000e-06 - loss: 0.0493 - UP: 13.33% UR:  0.01% UF:  0.03% P: 13.33% R:  0.01% F:  0.03%                                
2022-02-04 09:57:10 INFO dev:  loss: nan - UP:  0.00% UR:  0.00% UF:  0.00% P:  0.00% R:  0.00% F:  0.00%
2022-02-04 09:57:12 INFO test: loss: nan - UP:  0.00% UR:  0.00% UF:  0.00% P:  0.00% R:  0.00% F:  0.00%
2022-02-04 09:57:12 INFO 0:00:13.028747s elapsed

2022-02-04 09:57:12 INFO Epoch 3 / 200:
2022-02-04 09:57:22 INFO lr: 7.5000e-06 - loss: 0.0187 - UP: 43.49% UR:  0.83% UF:  1.63% P: 39.38% R:  0.75% F:  1.48%                                
2022-02-04 09:57:23 INFO dev:  loss: nan - UP: 45.87% UR:  1.10% UF:  2.16% P: 39.45% R:  0.95% F:  1.85%
2022-02-04 09:57:24 INFO test: loss: nan - UP: 45.87% UR:  1.10% UF:  2.16% P: 39.45% R:  0.95% F:  1.85%
2022-02-04 09:57:25 INFO 0:00:12.892630s elapsed (saved)

2022-02-04 09:57:25 INFO Epoch 4 / 200:
2022-02-04 09:57:35 INFO lr: 1.0000e-05 - loss: 0.0227 - UP: 41.19% UR:  3.64% UF:  6.68% P: 38.52% R:  3.40% F:  6.25%                                
2022-02-04 09:57:37 INFO dev:  loss: nan - UP: 39.94% UR:  2.89% UF:  5.39% P: 34.15% R:  2.47% F:  4.61%
2022-02-04 09:57:38 INFO test: loss: nan - UP: 40.12% UR:  2.91% UF:  5.43% P: 34.35% R:  2.49% F:  4.65%
2022-02-04 09:57:38 INFO 0:00:13.109289s elapsed (saved)

2022-02-04 09:57:38 INFO Epoch 5 / 200:
2022-02-04 09:57:49 INFO lr: 1.2500e-05 - loss: 0.0188 - UP: 40.59% UR:  7.42% UF: 12.55% P: 38.88% R:  7.11% F: 12.02%                                
2022-02-04 09:57:50 INFO dev:  loss: nan - UP: 60.76% UR:  3.18% UF:  6.04% P: 56.54% R:  2.96% F:  5.62%
2022-02-04 09:57:51 INFO test: loss: nan - UP: 60.92% UR:  3.20% UF:  6.08% P: 56.72% R:  2.98% F:  5.66%
2022-02-04 09:57:52 INFO 0:00:12.996236s elapsed (saved)

2022-02-04 09:57:52 INFO Epoch 6 / 200:
2022-02-04 09:58:02 INFO lr: 1.5000e-05 - loss: 0.0253 - UP: 49.43% UR: 10.85% UF: 17.80% P: 47.91% R: 10.52% F: 17.25%                                
2022-02-04 09:58:03 INFO dev:  loss: nan - UP: 34.57% UR: 17.30% UF: 23.06% P: 32.76% R: 16.40% F: 21.86%
2022-02-04 09:58:05 INFO test: loss: nan - UP: 34.58% UR: 17.33% UF: 23.08% P: 32.78% R: 16.42% F: 21.88%
2022-02-04 09:58:05 INFO 0:00:13.022203s elapsed (saved)

2022-02-04 09:58:05 INFO Epoch 7 / 200:
2022-02-04 09:58:16 INFO lr: 1.7500e-05 - loss: 0.0430 - UP: 49.85% UR: 14.49% UF: 22.45% P: 48.62% R: 14.13% F: 21.90%                                
2022-02-04 09:58:17 INFO dev:  loss: nan - UP: 59.68% UR:  8.23% UF: 14.47% P: 58.08% R:  8.01% F: 14.08%
2022-02-04 09:58:18 INFO test: loss: nan - UP: 59.71% UR:  8.21% UF: 14.44% P: 58.11% R:  7.99% F: 14.05%
2022-02-04 09:58:18 INFO 0:00:13.181216s elapsed

2022-02-04 09:58:18 INFO Epoch 8 / 200:
2022-02-04 09:58:29 INFO lr: 2.0000e-05 - loss: 0.0200 - UP: 57.33% UR: 21.29% UF: 31.05% P: 56.45% R: 20.96% F: 30.57%                                
2022-02-04 09:58:30 INFO dev:  loss: nan - UP: 47.83% UR:  8.52% UF: 14.46% P: 45.85% R:  8.17% F: 13.86%
2022-02-04 09:58:31 INFO test: loss: nan - UP: 47.83% UR:  8.52% UF: 14.46% P: 45.85% R:  8.17% F: 13.86%
2022-02-04 09:58:31 INFO 0:00:13.024829s elapsed

2022-02-04 09:58:31 INFO Epoch 9 / 200:
2022-02-04 09:58:42 INFO lr: 2.2500e-05 - loss: 0.0333 - UP: 52.60% UR: 21.07% UF: 30.09% P: 51.77% R: 20.74% F: 29.61%                                
2022-02-04 09:58:43 INFO dev:  loss: nan - UP: 62.37% UR:  6.84% UF: 12.33% P: 57.95% R:  6.36% F: 11.46%
2022-02-04 09:58:44 INFO test: loss: nan - UP: 62.45% UR:  6.86% UF: 12.37% P: 57.83% R:  6.36% F: 11.45%
2022-02-04 09:58:44 INFO 0:00:13.074168s elapsed

2022-02-04 09:58:44 INFO Epoch 10 / 200:
2022-02-04 09:58:55 INFO lr: 2.5000e-05 - loss: 0.0166 - UP: 56.10% UR: 21.60% UF: 31.19% P: 55.45% R: 21.35% F: 30.83%                                
2022-02-04 09:58:56 INFO dev:  loss: nan - UP: 51.84% UR: 14.61% UF: 22.80% P: 48.55% R: 13.68% F: 21.35%
2022-02-04 09:58:57 INFO test: loss: nan - UP: 51.84% UR: 14.61% UF: 22.80% P: 48.55% R: 13.68% F: 21.35%
2022-02-04 09:58:57 INFO 0:00:13.014130s elapsed

2022-02-04 09:58:57 INFO Epoch 11 / 200:
2022-02-04 09:59:08 INFO lr: 2.7500e-05 - loss: 0.0145 - UP: 58.12% UR: 29.69% UF: 39.30% P: 57.57% R: 29.40% F: 38.93%                                
2022-02-04 09:59:09 INFO dev:  loss: nan - UP: 39.97% UR: 17.99% UF: 24.81% P: 37.27% R: 16.77% F: 23.14%
2022-02-04 09:59:11 INFO test: loss: nan - UP: 40.01% UR: 17.99% UF: 24.82% P: 37.31% R: 16.77% F: 23.14%
2022-02-04 09:59:11 INFO 0:00:13.173265s elapsed (saved)

2022-02-04 09:59:11 INFO Epoch 12 / 200:
2022-02-04 09:59:22 INFO lr: 3.0000e-05 - loss: 0.0392 - UP: 58.94% UR: 30.22% UF: 39.95% P: 58.44% R: 29.96% F: 39.61%                                
2022-02-04 09:59:23 INFO dev:  loss: nan - UP: 41.84% UR: 26.37% UF: 32.35% P: 39.46% R: 24.87% F: 30.51%
2022-02-04 09:59:24 INFO test: loss: nan - UP: 41.82% UR: 26.35% UF: 32.33% P: 39.44% R: 24.85% F: 30.49%
2022-02-04 09:59:24 INFO 0:00:13.076202s elapsed (saved)

2022-02-04 09:59:24 INFO Epoch 13 / 200:
2022-02-04 09:59:35 INFO lr: 3.2500e-05 - loss: 0.0106 - UP: 61.43% UR: 31.54% UF: 41.68% P: 60.84% R: 31.24% F: 41.28%                                
2022-02-04 09:59:36 INFO dev:  loss: nan - UP: 50.61% UR: 11.92% UF: 19.29% P: 45.45% R: 10.70% F: 17.33%
2022-02-04 09:59:38 INFO test: loss: nan - UP: 50.61% UR: 11.92% UF: 19.29% P: 45.45% R: 10.70% F: 17.33%
2022-02-04 09:59:38 INFO 0:00:13.097968s elapsed

2022-02-04 09:59:38 INFO Epoch 14 / 200:
2022-02-04 09:59:48 INFO lr: 3.5000e-05 - loss: 0.0981 - UP: 58.87% UR: 30.31% UF: 40.02% P: 58.25% R: 29.99% F: 39.60%                                
2022-02-04 09:59:49 INFO dev:  loss: nan - UP: 45.71% UR: 19.86% UF: 27.69% P: 43.58% R: 18.94% F: 26.40%
2022-02-04 09:59:51 INFO test: loss: nan - UP: 45.73% UR: 19.86% UF: 27.70% P: 43.60% R: 18.94% F: 26.40%
2022-02-04 09:59:51 INFO 0:00:13.112089s elapsed

2022-02-04 09:59:51 INFO Epoch 15 / 200:
2022-02-04 10:00:01 INFO lr: 3.7500e-05 - loss: 0.1007 - UP: 61.66% UR: 33.17% UF: 43.13% P: 61.05% R: 32.84% F: 42.71%                                
2022-02-04 10:00:02 INFO dev:  loss: nan - UP: 45.20% UR: 24.85% UF: 32.07% P: 43.76% R: 24.06% F: 31.05%
2022-02-04 10:00:04 INFO test: loss: nan - UP: 45.18% UR: 24.85% UF: 32.07% P: 43.74% R: 24.06% F: 31.04%
2022-02-04 10:00:04 INFO 0:00:13.085366s elapsed (saved)

2022-02-04 10:00:04 INFO Epoch 16 / 200:
2022-02-04 10:00:15 INFO lr: 4.0000e-05 - loss: 0.0117 - UP: 63.55% UR: 33.30% UF: 43.70% P: 62.94% R: 32.98% F: 43.28%                                
2022-02-04 10:00:16 INFO dev:  loss: nan - UP: 50.42% UR: 18.49% UF: 27.06% P: 48.98% R: 17.97% F: 26.29%
2022-02-04 10:00:17 INFO test: loss: nan - UP: 50.45% UR: 18.49% UF: 27.07% P: 49.01% R: 17.97% F: 26.29%
2022-02-04 10:00:17 INFO 0:00:13.067962s elapsed

2022-02-04 10:00:17 INFO Epoch 17 / 200:
2022-02-04 10:00:28 INFO lr: 4.2500e-05 - loss: 0.0298 - UP: 61.63% UR: 35.19% UF: 44.80% P: 61.18% R: 34.94% F: 44.48%                                
2022-02-04 10:00:29 INFO dev:  loss: nan - UP: 49.79% UR: 24.01% UF: 32.40% P: 47.46% R: 22.89% F: 30.88%
2022-02-04 10:00:30 INFO test: loss: nan - UP: 49.82% UR: 24.01% UF: 32.41% P: 47.48% R: 22.89% F: 30.89%
2022-02-04 10:00:30 INFO 0:00:12.938851s elapsed

2022-02-04 10:00:30 INFO Epoch 18 / 200:
2022-02-04 10:00:41 INFO lr: 4.5000e-05 - loss: 0.0073 - UP: 69.24% UR: 43.83% UF: 53.68% P: 68.74% R: 43.51% F: 53.29%                                
2022-02-04 10:00:42 INFO dev:  loss: nan - UP: 49.57% UR: 29.31% UF: 36.84% P: 46.66% R: 27.59% F: 34.67%
2022-02-04 10:00:43 INFO test: loss: nan - UP: 49.55% UR: 29.29% UF: 36.82% P: 46.64% R: 27.57% F: 34.65%
2022-02-04 10:00:44 INFO 0:00:12.990013s elapsed (saved)

2022-02-04 10:00:44 INFO Epoch 19 / 200:
2022-02-04 10:00:54 INFO lr: 4.7500e-05 - loss: 0.0131 - UP: 72.06% UR: 51.07% UF: 59.78% P: 71.81% R: 50.89% F: 59.57%                                
2022-02-04 10:00:55 INFO dev:  loss: nan - UP: 46.16% UR: 24.96% UF: 32.40% P: 43.31% R: 23.42% F: 30.40%
2022-02-04 10:00:57 INFO test: loss: nan - UP: 46.16% UR: 24.96% UF: 32.40% P: 43.31% R: 23.42% F: 30.40%
2022-02-04 10:00:57 INFO 0:00:13.034172s elapsed

2022-02-04 10:00:57 INFO Epoch 20 / 200:
2022-02-04 10:01:07 INFO lr: 5.0000e-05 - loss: 0.0076 - UP: 67.47% UR: 44.87% UF: 53.90% P: 67.20% R: 44.70% F: 53.69%                                
2022-02-04 10:01:09 INFO dev:  loss: nan - UP: 52.89% UR: 15.96% UF: 24.52% P: 50.77% R: 15.32% F: 23.53%
2022-02-04 10:01:10 INFO test: loss: nan - UP: 52.86% UR: 15.93% UF: 24.49% P: 50.73% R: 15.29% F: 23.50%
2022-02-04 10:01:10 INFO 0:00:13.093453s elapsed

2022-02-04 10:01:10 INFO Epoch 21 / 200:
2022-02-04 10:01:20 INFO lr: 4.9722e-05 - loss: 0.0119 - UP: 69.15% UR: 42.68% UF: 52.78% P: 68.78% R: 42.45% F: 52.50%                                
2022-02-04 10:01:22 INFO dev:  loss: nan - UP: 40.25% UR: 32.38% UF: 35.89% P: 38.79% R: 31.21% F: 34.59%
2022-02-04 10:01:23 INFO test: loss: nan - UP: 40.23% UR: 32.35% UF: 35.87% P: 38.78% R: 31.19% F: 34.57%
2022-02-04 10:01:23 INFO 0:00:13.161000s elapsed

2022-02-04 10:01:23 INFO Epoch 22 / 200:
2022-02-04 10:01:33 INFO lr: 4.9444e-05 - loss: 0.0018 - UP: 70.47% UR: 49.32% UF: 58.03% P: 70.07% R: 49.05% F: 57.71%                                
2022-02-04 10:01:35 INFO dev:  loss: nan - UP: 43.90% UR: 32.49% UF: 37.34% P: 42.08% R: 31.14% F: 35.79%
2022-02-04 10:01:36 INFO test: loss: nan - UP: 43.90% UR: 32.47% UF: 37.33% P: 42.08% R: 31.12% F: 35.78%
2022-02-04 10:01:36 INFO 0:00:13.049294s elapsed (saved)

2022-02-04 10:01:36 INFO Epoch 23 / 200:
2022-02-04 10:01:47 INFO lr: 4.9167e-05 - loss: 0.0017 - UP: 74.37% UR: 55.51% UF: 63.57% P: 74.09% R: 55.30% F: 63.33%                                
2022-02-04 10:01:48 INFO dev:  loss: nan - UP: 49.57% UR: 30.77% UF: 37.97% P: 47.83% R: 29.68% F: 36.63%
2022-02-04 10:01:49 INFO test: loss: nan - UP: 49.57% UR: 30.79% UF: 37.99% P: 47.83% R: 29.71% F: 36.65%
2022-02-04 10:01:50 INFO 0:00:12.979300s elapsed (saved)

2022-02-04 10:01:50 INFO Epoch 24 / 200:
2022-02-04 10:02:00 INFO lr: 4.8889e-05 - loss: 0.0058 - UP: 78.90% UR: 62.56% UF: 69.79% P: 78.76% R: 62.45% F: 69.66%                                
2022-02-04 10:02:02 INFO dev:  loss: nan - UP: 46.37% UR: 38.12% UF: 41.84% P: 44.31% R: 36.42% F: 39.98%
2022-02-04 10:02:03 INFO test: loss: nan - UP: 46.36% UR: 38.09% UF: 41.82% P: 44.29% R: 36.39% F: 39.96%
2022-02-04 10:02:03 INFO 0:00:13.196581s elapsed (saved)

2022-02-04 10:02:03 INFO Epoch 25 / 200:
2022-02-04 10:02:14 INFO lr: 4.8611e-05 - loss: 0.0208 - UP: 81.95% UR: 70.25% UF: 75.65% P: 81.80% R: 70.12% F: 75.51%                                
2022-02-04 10:02:15 INFO dev:  loss: nan - UP: 42.88% UR: 39.02% UF: 40.86% P: 40.87% R: 37.19% F: 38.94%
2022-02-04 10:02:16 INFO test: loss: nan - UP: 42.90% UR: 39.04% UF: 40.88% P: 40.88% R: 37.21% F: 38.96%
2022-02-04 10:02:16 INFO 0:00:13.051098s elapsed

2022-02-04 10:02:16 INFO Epoch 26 / 200:
2022-02-04 10:02:27 INFO lr: 4.8333e-05 - loss: 0.0155 - UP: 76.84% UR: 60.17% UF: 67.49% P: 76.72% R: 60.08% F: 67.39%                                
2022-02-04 10:02:28 INFO dev:  loss: nan - UP: 52.30% UR: 20.30% UF: 29.25% P: 49.86% R: 19.36% F: 27.89%
2022-02-04 10:02:30 INFO test: loss: nan - UP: 52.30% UR: 20.33% UF: 29.28% P: 49.86% R: 19.38% F: 27.91%
2022-02-04 10:02:30 INFO 0:00:13.121767s elapsed

2022-02-04 10:02:30 INFO Epoch 27 / 200:
2022-02-04 10:02:40 INFO lr: 4.8056e-05 - loss: 0.0065 - UP: 73.62% UR: 55.20% UF: 63.10% P: 73.38% R: 55.02% F: 62.89%                                
2022-02-04 10:02:41 INFO dev:  loss: nan - UP: 37.44% UR: 36.66% UF: 37.04% P: 36.17% R: 35.42% F: 35.79%
2022-02-04 10:02:43 INFO test: loss: nan - UP: 37.43% UR: 36.68% UF: 37.05% P: 36.17% R: 35.44% F: 35.80%
2022-02-04 10:02:43 INFO 0:00:13.105290s elapsed

2022-02-04 10:02:43 INFO Epoch 28 / 200:
2022-02-04 10:02:53 INFO lr: 4.7778e-05 - loss: nan - UP: 75.87% UR: 57.94% UF: 65.70% P: 75.65% R: 57.78% F: 65.52%                                   
2022-02-04 10:02:55 INFO dev:  loss: nan - UP: 52.03% UR: 30.59% UF: 38.53% P: 49.70% R: 29.22% F: 36.80%
2022-02-04 10:02:56 INFO test: loss: nan - UP: 52.03% UR: 30.61% UF: 38.54% P: 49.70% R: 29.24% F: 36.82%
2022-02-04 10:02:56 INFO 0:00:13.188228s elapsed

2022-02-04 10:02:56 INFO Epoch 29 / 200:
2022-02-04 10:03:06 INFO lr: 4.7500e-05 - loss: 0.0020 - UP: 79.52% UR: 62.90% UF: 70.24% P: 79.14% R: 62.61% F: 69.91%                                
2022-02-04 10:03:08 INFO dev:  loss: nan - UP: 43.59% UR: 39.62% UF: 41.51% P: 41.53% R: 37.74% F: 39.54%
2022-02-04 10:03:09 INFO test: loss: nan - UP: 43.58% UR: 39.62% UF: 41.50% P: 41.51% R: 37.74% F: 39.54%
2022-02-04 10:03:09 INFO 0:00:13.001304s elapsed

2022-02-04 10:03:09 INFO Epoch 30 / 200:
2022-02-04 10:03:19 INFO lr: 4.7222e-05 - loss: 0.0030 - UP: 80.91% UR: 69.33% UF: 74.67% P: 80.77% R: 69.21% F: 74.54%                                
2022-02-04 10:03:21 INFO dev:  loss: nan - UP: 51.54% UR: 30.94% UF: 38.67% P: 49.12% R: 29.49% F: 36.85%
2022-02-04 10:03:22 INFO test: loss: nan - UP: 51.58% UR: 30.94% UF: 38.68% P: 49.15% R: 29.49% F: 36.86%
2022-02-04 10:03:22 INFO 0:00:13.097030s elapsed

2022-02-04 10:03:22 INFO Epoch 31 / 200:
2022-02-04 10:03:33 INFO lr: 4.6944e-05 - loss: 0.0090 - UP: 78.55% UR: 63.56% UF: 70.26% P: 78.28% R: 63.33% F: 70.02%                                
2022-02-04 10:03:34 INFO dev:  loss: nan - UP: 50.13% UR: 33.57% UF: 40.21% P: 48.62% R: 32.55% F: 39.00%
2022-02-04 10:03:35 INFO test: loss: nan - UP: 50.15% UR: 33.57% UF: 40.22% P: 48.63% R: 32.55% F: 39.00%
2022-02-04 10:03:35 INFO 0:00:13.140595s elapsed

2022-02-04 10:03:35 INFO Epoch 32 / 200:
2022-02-04 10:03:46 INFO lr: 4.6667e-05 - loss: 0.0073 - UP: 80.64% UR: 66.99% UF: 73.18% P: 80.47% R: 66.85% F: 73.03%                                
2022-02-04 10:03:47 INFO dev:  loss: nan - UP: 51.14% UR: 28.16% UF: 36.32% P: 48.34% R: 26.62% F: 34.33%
2022-02-04 10:03:48 INFO test: loss: nan - UP: 51.16% UR: 28.16% UF: 36.33% P: 48.36% R: 26.62% F: 34.33%
2022-02-04 10:03:48 INFO 0:00:13.094140s elapsed

2022-02-04 10:03:48 INFO Epoch 33 / 200:
2022-02-04 10:03:59 INFO lr: 4.6389e-05 - loss: 0.0053 - UP: 82.42% UR: 72.51% UF: 77.15% P: 82.36% R: 72.45% F: 77.09%                                
2022-02-04 10:04:00 INFO dev:  loss: nan - UP: 46.05% UR: 39.86% UF: 42.73% P: 43.70% R: 37.83% F: 40.55%
2022-02-04 10:04:01 INFO test: loss: nan - UP: 46.05% UR: 39.88% UF: 42.74% P: 43.71% R: 37.85% F: 40.57%
2022-02-04 10:04:02 INFO 0:00:13.056610s elapsed (saved)

2022-02-04 10:04:02 INFO Epoch 34 / 200:
2022-02-04 10:04:12 INFO lr: 4.6111e-05 - loss: 0.0119 - UP: 81.07% UR: 69.05% UF: 74.58% P: 80.78% R: 68.80% F: 74.31%                                
2022-02-04 10:04:13 INFO dev:  loss: nan - UP: 49.24% UR: 22.84% UF: 31.21% P: 46.43% R: 21.54% F: 29.43%
2022-02-04 10:04:15 INFO test: loss: nan - UP: 49.24% UR: 22.84% UF: 31.21% P: 46.43% R: 21.54% F: 29.43%
2022-02-04 10:04:15 INFO 0:00:13.023962s elapsed

2022-02-04 10:04:15 INFO Epoch 35 / 200:
2022-02-04 10:04:25 INFO lr: 4.5833e-05 - loss: 0.0035 - UP: 79.40% UR: 63.52% UF: 70.58% P: 79.23% R: 63.39% F: 70.43%                                
2022-02-04 10:04:26 INFO dev:  loss: nan - UP: 41.78% UR: 38.60% UF: 40.13% P: 39.75% R: 36.72% F: 38.18%
2022-02-04 10:04:28 INFO test: loss: nan - UP: 41.77% UR: 38.60% UF: 40.12% P: 39.74% R: 36.72% F: 38.17%
2022-02-04 10:04:28 INFO 0:00:13.049699s elapsed

2022-02-04 10:04:28 INFO Epoch 36 / 200:
2022-02-04 10:04:38 INFO lr: 4.5556e-05 - loss: 0.0074 - UP: 83.22% UR: 71.91% UF: 77.15% P: 83.12% R: 71.83% F: 77.06%                                
2022-02-04 10:04:40 INFO dev:  loss: nan - UP: 55.41% UR: 26.24% UF: 35.61% P: 53.45% R: 25.31% F: 34.36%
2022-02-04 10:04:41 INFO test: loss: nan - UP: 55.43% UR: 26.24% UF: 35.62% P: 53.47% R: 25.31% F: 34.36%
2022-02-04 10:04:41 INFO 0:00:13.153910s elapsed

2022-02-04 10:04:41 INFO Epoch 37 / 200:
2022-02-04 10:04:51 INFO lr: 4.5278e-05 - loss: 0.0074 - UP: 83.69% UR: 74.48% UF: 78.82% P: 83.49% R: 74.30% F: 78.63%                                
2022-02-04 10:04:53 INFO dev:  loss: nan - UP: 51.00% UR: 33.15% UF: 40.18% P: 49.24% R: 32.00% F: 38.79%
2022-02-04 10:04:54 INFO test: loss: nan - UP: 51.00% UR: 33.15% UF: 40.18% P: 49.24% R: 32.00% F: 38.79%
2022-02-04 10:04:54 INFO 0:00:13.114914s elapsed

2022-02-04 10:04:54 INFO Epoch 38 / 200:
2022-02-04 10:05:04 INFO lr: 4.5000e-05 - loss: 0.0020 - UP: 87.57% UR: 81.20% UF: 84.26% P: 87.51% R: 81.14% F: 84.20%                                
2022-02-04 10:05:06 INFO dev:  loss: nan - UP: 54.72% UR: 32.00% UF: 40.38% P: 52.98% R: 30.99% F: 39.10%
2022-02-04 10:05:07 INFO test: loss: nan - UP: 54.70% UR: 32.00% UF: 40.38% P: 52.96% R: 30.99% F: 39.10%
2022-02-04 10:05:07 INFO 0:00:13.068082s elapsed

2022-02-04 10:05:07 INFO Epoch 39 / 200:
2022-02-04 10:05:18 INFO lr: 4.4722e-05 - loss: 0.0018 - UP: 90.65% UR: 86.63% UF: 88.59% P: 90.65% R: 86.63% F: 88.59%                                
2022-02-04 10:05:19 INFO dev:  loss: nan - UP: 53.68% UR: 33.61% UF: 41.34% P: 51.57% R: 32.29% F: 39.71%
2022-02-04 10:05:20 INFO test: loss: nan - UP: 53.74% UR: 33.61% UF: 41.36% P: 51.62% R: 32.29% F: 39.73%
2022-02-04 10:05:20 INFO 0:00:13.099658s elapsed

2022-02-04 10:05:20 INFO Epoch 40 / 200:
2022-02-04 10:05:31 INFO lr: 4.4444e-05 - loss: 0.0028 - UP: 92.91% UR: 90.33% UF: 91.60% P: 92.91% R: 90.33% F: 91.60%                                
2022-02-04 10:05:32 INFO dev:  loss: nan - UP: 56.00% UR: 34.61% UF: 42.78% P: 53.96% R: 33.35% F: 41.22%
2022-02-04 10:05:33 INFO test: loss: nan - UP: 56.02% UR: 34.61% UF: 42.78% P: 53.95% R: 33.33% F: 41.20%
2022-02-04 10:05:34 INFO 0:00:13.171876s elapsed (saved)

2022-02-04 10:05:34 INFO Epoch 41 / 200:
2022-02-04 10:05:44 INFO lr: 4.4167e-05 - loss: 0.0030 - UP: 94.47% UR: 91.99% UF: 93.21% P: 94.47% R: 91.99% F: 93.21%                                
2022-02-04 10:05:46 INFO dev:  loss: nan - UP: 51.81% UR: 39.13% UF: 44.59% P: 49.88% R: 37.67% F: 42.93%
2022-02-04 10:05:47 INFO test: loss: nan - UP: 51.81% UR: 39.13% UF: 44.59% P: 49.88% R: 37.67% F: 42.93%
2022-02-04 10:05:47 INFO 0:00:13.101104s elapsed (saved)

2022-02-04 10:05:47 INFO Epoch 42 / 200:
2022-02-04 10:05:58 INFO lr: 4.3889e-05 - loss: 0.0207 - UP: 95.02% UR: 93.58% UF: 94.29% P: 95.02% R: 93.58% F: 94.29%                                
2022-02-04 10:05:59 INFO dev:  loss: nan - UP: 57.45% UR: 28.01% UF: 37.66% P: 55.77% R: 27.19% F: 36.56%
2022-02-04 10:06:00 INFO test: loss: nan - UP: 57.45% UR: 28.01% UF: 37.66% P: 55.77% R: 27.19% F: 36.56%
2022-02-04 10:06:00 INFO 0:00:13.038488s elapsed

2022-02-04 10:06:00 INFO Epoch 43 / 200:
2022-02-04 10:06:11 INFO lr: 4.3611e-05 - loss: 0.0025 - UP: 95.58% UR: 93.92% UF: 94.74% P: 95.57% R: 93.91% F: 94.73%                                
2022-02-04 10:06:12 INFO dev:  loss: nan - UP: 58.25% UR: 31.16% UF: 40.60% P: 56.06% R: 29.99% F: 39.08%
2022-02-04 10:06:13 INFO test: loss: nan - UP: 58.23% UR: 31.16% UF: 40.60% P: 56.04% R: 29.99% F: 39.07%
2022-02-04 10:06:13 INFO 0:00:13.081608s elapsed

2022-02-04 10:06:13 INFO Epoch 44 / 200:
2022-02-04 10:06:24 INFO lr: 4.3333e-05 - loss: 0.0000 - UP: 96.23% UR: 95.58% UF: 95.90% P: 96.23% R: 95.58% F: 95.90%                                
2022-02-04 10:06:25 INFO dev:  loss: nan - UP: 51.77% UR: 37.54% UF: 43.52% P: 50.03% R: 36.28% F: 42.06%
2022-02-04 10:06:27 INFO test: loss: nan - UP: 51.75% UR: 37.54% UF: 43.51% P: 50.02% R: 36.28% F: 42.06%
2022-02-04 10:06:27 INFO 0:00:13.319029s elapsed

2022-02-04 10:06:27 INFO Epoch 45 / 200:
2022-02-04 10:06:37 INFO lr: 4.3056e-05 - loss: 0.0006 - UP: 96.99% UR: 95.80% UF: 96.39% P: 96.99% R: 95.80% F: 96.39%                                
2022-02-04 10:06:39 INFO dev:  loss: nan - UP: 52.07% UR: 37.83% UF: 43.82% P: 50.24% R: 36.50% F: 42.29%
2022-02-04 10:06:40 INFO test: loss: nan - UP: 52.05% UR: 37.83% UF: 43.81% P: 50.23% R: 36.50% F: 42.28%
2022-02-04 10:06:40 INFO 0:00:13.106479s elapsed

2022-02-04 10:06:40 INFO Epoch 46 / 200:
2022-02-04 10:06:50 INFO lr: 4.2778e-05 - loss: 0.0015 - UP: 96.99% UR: 96.60% UF: 96.79% P: 96.99% R: 96.60% F: 96.79%                                
2022-02-04 10:06:52 INFO dev:  loss: nan - UP: 57.19% UR: 31.76% UF: 40.84% P: 55.05% R: 30.57% F: 39.31%
2022-02-04 10:06:53 INFO test: loss: nan - UP: 57.17% UR: 31.76% UF: 40.83% P: 55.03% R: 30.57% F: 39.30%
2022-02-04 10:06:53 INFO 0:00:13.048740s elapsed

2022-02-04 10:06:53 INFO Epoch 47 / 200:
2022-02-04 10:07:03 INFO lr: 4.2500e-05 - loss: 0.0007 - UP: 97.78% UR: 96.94% UF: 97.36% P: 97.78% R: 96.94% F: 97.36%                                
2022-02-04 10:07:05 INFO dev:  loss: nan - UP: 50.68% UR: 37.67% UF: 43.22% P: 49.11% R: 36.50% F: 41.88%
2022-02-04 10:07:06 INFO test: loss: nan - UP: 50.68% UR: 37.67% UF: 43.22% P: 49.11% R: 36.50% F: 41.88%
2022-02-04 10:07:06 INFO 0:00:13.176287s elapsed

2022-02-04 10:07:06 INFO Epoch 48 / 200:
2022-02-04 10:07:17 INFO lr: 4.2222e-05 - loss: 0.0006 - UP: 97.92% UR: 97.44% UF: 97.68% P: 97.92% R: 97.44% F: 97.68%                                
2022-02-04 10:07:18 INFO dev:  loss: nan - UP: 57.31% UR: 33.99% UF: 42.67% P: 55.34% R: 32.82% F: 41.20%
2022-02-04 10:07:19 INFO test: loss: nan - UP: 57.31% UR: 33.99% UF: 42.67% P: 55.34% R: 32.82% F: 41.20%
2022-02-04 10:07:19 INFO 0:00:13.136762s elapsed

2022-02-04 10:07:19 INFO Epoch 49 / 200:
2022-02-04 10:07:30 INFO lr: 4.1944e-05 - loss: nan - UP: 95.75% UR: 95.01% UF: 95.38% P: 95.75% R: 95.01% F: 95.38%                                   
2022-02-04 10:07:31 INFO dev:  loss: nan - UP: 45.90% UR: 39.09% UF: 42.22% P: 44.38% R: 37.78% F: 40.82%
2022-02-04 10:07:32 INFO test: loss: nan - UP: 45.93% UR: 39.09% UF: 42.23% P: 44.40% R: 37.78% F: 40.83%
2022-02-04 10:07:32 INFO 0:00:13.107475s elapsed

2022-02-04 10:07:32 INFO Epoch 50 / 200:
2022-02-04 10:07:43 INFO lr: 4.1667e-05 - loss: 0.1071 - UP: 84.05% UR: 76.39% UF: 80.03% P: 83.86% R: 76.22% F: 79.86%                                
2022-02-04 10:07:44 INFO dev:  loss: nan - UP: 19.35% UR: 14.43% UF: 16.53% P: 16.42% R: 12.25% F: 14.03%
2022-02-04 10:07:45 INFO test: loss: nan - UP: 19.37% UR: 14.43% UF: 16.54% P: 16.43% R: 12.25% F: 14.04%
2022-02-04 10:07:45 INFO 0:00:13.030685s elapsed

2022-02-04 10:07:45 INFO Epoch 51 / 200:
2022-02-04 10:07:56 INFO lr: 4.1389e-05 - loss: 0.0079 - UP: 61.44% UR: 33.07% UF: 43.00% P: 60.83% R: 32.75% F: 42.58%                                
2022-02-04 10:07:57 INFO dev:  loss: nan - UP: 40.69% UR: 26.29% UF: 31.94% P: 39.05% R: 25.23% F: 30.65%
2022-02-04 10:07:58 INFO test: loss: nan - UP: 40.72% UR: 26.29% UF: 31.95% P: 39.08% R: 25.23% F: 30.66%
2022-02-04 10:07:58 INFO 0:00:12.984066s elapsed

2022-02-04 10:07:58 INFO Epoch 52 / 200:
2022-02-04 10:08:09 INFO lr: 4.1111e-05 - loss: nan - UP: 78.06% UR: 56.56% UF: 65.59% P: 77.72% R: 56.31% F: 65.30%                                   
2022-02-04 10:08:10 INFO dev:  loss: nan - UP: 52.43% UR: 17.17% UF: 25.87% P: 50.61% R: 16.57% F: 24.97%
2022-02-04 10:08:11 INFO test: loss: nan - UP: 52.43% UR: 17.17% UF: 25.87% P: 50.61% R: 16.57% F: 24.97%
2022-02-04 10:08:11 INFO 0:00:13.154260s elapsed

2022-02-04 10:08:11 INFO Epoch 53 / 200:
2022-02-04 10:08:22 INFO lr: 4.0833e-05 - loss: 0.0029 - UP: 81.42% UR: 71.11% UF: 75.92% P: 81.35% R: 71.05% F: 75.85%                                
2022-02-04 10:08:23 INFO dev:  loss: nan - UP: 58.30% UR: 26.90% UF: 36.82% P: 55.52% R: 25.62% F: 35.06%
2022-02-04 10:08:25 INFO test: loss: nan - UP: 58.30% UR: 26.90% UF: 36.82% P: 55.52% R: 25.62% F: 35.06%
2022-02-04 10:08:25 INFO 0:00:13.084500s elapsed

2022-02-04 10:08:25 INFO Epoch 54 / 200:
2022-02-04 10:08:35 INFO lr: 4.0556e-05 - loss: 0.0019 - UP: 89.07% UR: 82.06% UF: 85.42% P: 89.05% R: 82.05% F: 85.41%                                
2022-02-04 10:08:36 INFO dev:  loss: nan - UP: 58.92% UR: 30.83% UF: 40.48% P: 57.19% R: 29.93% F: 39.29%
2022-02-04 10:08:37 INFO test: loss: nan - UP: 58.95% UR: 30.83% UF: 40.49% P: 57.22% R: 29.93% F: 39.30%
2022-02-04 10:08:37 INFO 0:00:12.979073s elapsed

2022-02-04 10:08:37 INFO Epoch 55 / 200:
2022-02-04 10:08:48 INFO lr: 4.0278e-05 - loss: 0.0003 - UP: 91.65% UR: 88.26% UF: 89.92% P: 91.62% R: 88.23% F: 89.90%                                
2022-02-04 10:08:49 INFO dev:  loss: nan - UP: 54.82% UR: 33.50% UF: 41.59% P: 53.12% R: 32.47% F: 40.30%
2022-02-04 10:08:50 INFO test: loss: nan - UP: 54.84% UR: 33.50% UF: 41.59% P: 53.14% R: 32.47% F: 40.31%
2022-02-04 10:08:50 INFO 0:00:12.878954s elapsed

2022-02-04 10:08:50 INFO Epoch 56 / 200:
2022-02-04 10:09:01 INFO lr: 4.0000e-05 - loss: 0.0025 - UP: 93.45% UR: 91.93% UF: 92.68% P: 93.43% R: 91.91% F: 92.66%                                
2022-02-04 10:09:02 INFO dev:  loss: nan - UP: 51.94% UR: 36.90% UF: 43.15% P: 50.26% R: 35.71% F: 41.75%
2022-02-04 10:09:03 INFO test: loss: nan - UP: 51.94% UR: 36.88% UF: 43.13% P: 50.26% R: 35.69% F: 41.74%
2022-02-04 10:09:03 INFO 0:00:13.024973s elapsed

2022-02-04 10:09:03 INFO Epoch 57 / 200:
2022-02-04 10:09:14 INFO lr: 3.9722e-05 - loss: 0.0014 - UP: 94.97% UR: 93.19% UF: 94.07% P: 94.97% R: 93.18% F: 94.07%                                
2022-02-04 10:09:15 INFO dev:  loss: nan - UP: 51.66% UR: 36.02% UF: 42.44% P: 49.54% R: 34.54% F: 40.70%
2022-02-04 10:09:16 INFO test: loss: nan - UP: 51.65% UR: 36.00% UF: 42.42% P: 49.53% R: 34.52% F: 40.68%
2022-02-04 10:09:16 INFO 0:00:12.951972s elapsed

2022-02-04 10:09:16 INFO Epoch 58 / 200:
2022-02-04 10:09:27 INFO lr: 3.9444e-05 - loss: 0.0015 - UP: 95.68% UR: 94.54% UF: 95.10% P: 95.66% R: 94.52% F: 95.08%                                
2022-02-04 10:09:28 INFO dev:  loss: nan - UP: 49.23% UR: 38.87% UF: 43.44% P: 47.11% R: 37.19% F: 41.56%
2022-02-04 10:09:29 INFO test: loss: nan - UP: 49.26% UR: 38.87% UF: 43.45% P: 47.13% R: 37.19% F: 41.57%
2022-02-04 10:09:29 INFO 0:00:13.019302s elapsed

2022-02-04 10:09:29 INFO Epoch 59 / 200:
2022-02-04 10:09:40 INFO lr: 3.9167e-05 - loss: 0.0010 - UP: 93.66% UR: 92.41% UF: 93.03% P: 93.56% R: 92.31% F: 92.93%                                
2022-02-04 10:09:41 INFO dev:  loss: nan - UP: 49.95% UR: 35.62% UF: 41.59% P: 48.16% R: 34.34% F: 40.09%
2022-02-04 10:09:42 INFO test: loss: nan - UP: 49.97% UR: 35.64% UF: 41.61% P: 48.17% R: 34.36% F: 40.11%
2022-02-04 10:09:42 INFO 0:00:13.093973s elapsed

Is it a rounding error or a bug?

Regards.

yzhangcs commented 2 years ago

@MinionAttack Hi, have you figured out the issue? I just trained a biaffine-sdp model using albert-base-v2, and the results appeared to be normal.

2022-02-23 14:00:28 INFO BiaffineSemanticDependencyModel(
  (encoder): TransformerEmbedding(albert-base-v2, n_layers=4, n_out=768, stride=256, pooling=mean, pad_index=0, requires_grad=True)
  (encoder_dropout): Dropout(p=0.1, inplace=False)
  (edge_mlp_d): MLP(n_in=768, n_out=600, dropout=0.25)
  (edge_mlp_h): MLP(n_in=768, n_out=600, dropout=0.25)
  (label_mlp_d): MLP(n_in=768, n_out=600, dropout=0.33)
  (label_mlp_h): MLP(n_in=768, n_out=600, dropout=0.33)
  (edge_attn): Biaffine(n_in=600, n_out=2, bias_x=True, bias_y=True)
  (label_attn): Biaffine(n_in=600, n_out=60, bias_x=True, bias_y=True)
  (criterion): CrossEntropyLoss()
)

2022-02-23 14:00:28 INFO Loading the data
2022-02-23 14:01:32 INFO                                         
train: Dataset(n_sentences=33916, n_batches=322, n_buckets=32)
dev:   Dataset(n_sentences=1692, n_batches=33, n_buckets=32)
test:  Dataset(n_sentences=1692, n_batches=32, n_buckets=32)

2022-02-23 14:01:32 INFO Epoch 1 / 10:
2022-02-23 14:03:10 INFO lr: 5.0000e-05 - loss: 0.6832 - UP: 51.01% UR: 35.75% UF: 42.04% P: 47.96% R: 33.61% F: 39.52%                                
2022-02-23 14:03:11 INFO dev:  loss: 0.2957 - UP: 21.90% UR: 87.71% UF: 35.05% P: 20.66% R: 82.74% F: 33.06%
2022-02-23 14:03:13 INFO test: loss: 0.2911 - UP: 21.90% UR: 87.71% UF: 35.05% P: 20.66% R: 82.74% F: 33.06%
2022-02-23 14:03:16 INFO 0:01:40.838080s elapsed (saved)

2022-02-23 14:03:16 INFO Epoch 2 / 10:
2022-02-23 14:04:53 INFO lr: 4.4444e-05 - loss: 0.0169 - UP: 75.56% UR: 67.14% UF: 71.10% P: 73.67% R: 65.46% F: 69.32%                                
2022-02-23 14:04:55 INFO dev:  loss: 0.0547 - UP: 87.28% UR: 81.35% UF: 84.21% P: 85.79% R: 79.96% F: 82.77%
2022-02-23 14:04:57 INFO test: loss: 0.0519 - UP: 87.28% UR: 81.35% UF: 84.21% P: 85.79% R: 79.96% F: 82.77%
2022-02-23 14:04:59 INFO 0:01:41.048991s elapsed (saved)

2022-02-23 14:04:59 INFO Epoch 3 / 10:
2022-02-23 14:06:36 INFO lr: 3.8889e-05 - loss: 0.0143 - UP: 84.24% UR: 76.87% UF: 80.39% P: 82.88% R: 75.63% F: 79.09%                                
2022-02-23 14:06:38 INFO dev:  loss: 0.0463 - UP: 87.70% UR: 83.84% UF: 85.73% P: 86.49% R: 82.68% F: 84.54%
2022-02-23 14:06:40 INFO test: loss: 0.0431 - UP: 87.70% UR: 83.84% UF: 85.73% P: 86.49% R: 82.68% F: 84.54%
2022-02-23 14:06:42 INFO 0:01:40.983313s elapsed (saved)

2022-02-23 14:06:42 INFO Epoch 4 / 10:
2022-02-23 14:08:19 INFO lr: 3.3333e-05 - loss: 0.2192 - UP: 88.32% UR: 83.40% UF: 85.79% P: 87.28% R: 82.41% F: 84.78%                                
2022-02-23 14:08:21 INFO dev:  loss: 0.0485 - UP: 83.48% UR: 86.43% UF: 84.93% P: 82.34% R: 85.26% F: 83.77%
2022-02-23 14:08:23 INFO test: loss: 0.0459 - UP: 83.48% UR: 86.43% UF: 84.93% P: 82.34% R: 85.26% F: 83.77%
2022-02-23 14:08:23 INFO 0:01:40.920955s elapsed

2022-02-23 14:08:23 INFO Epoch 5 / 10:
2022-02-23 14:10:00 INFO lr: 2.7778e-05 - loss: 0.0097 - UP: 87.49% UR: 83.32% UF: 85.35% P: 86.56% R: 82.44% F: 84.45%                                
2022-02-23 14:10:02 INFO dev:  loss: 0.0443 - UP: 89.97% UR: 81.80% UF: 85.69% P: 88.89% R: 80.82% F: 84.66%
2022-02-23 14:10:04 INFO test: loss: 0.0402 - UP: 89.97% UR: 81.80% UF: 85.69% P: 88.89% R: 80.82% F: 84.66%
2022-02-23 14:10:06 INFO 0:01:41.001333s elapsed (saved)
MinionAttack commented 2 years ago

Nope, I haven't given it much importance, it happens randomly and maybe if you let it do more iterations it would appear. I want to think that they are rounding or decimal errors because it doesn't make sense to me that some epochs are right and in others it doesn't match.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 2 years ago

This issue was closed because it has been inactive for 7 days since being marked as stale.