Closed rootofmylife closed 2 years ago
@rootofmylife Yes, sorry it's indeed a bug. I have pushed the fix to main branch recently, but it has not been released yet.
@rootofmylife Please check this commit: https://github.com/yzhangcs/parser/commit/716ac3ed. The error occurs because I accidentally saved the model of last epoch rather than the best-performing one.
@yzhangcs Yeah, I have noticed that and updated to the latest code too. Please see my attached image.
@rootofmylife
P/S: I use the latest code and not change anything. Sorry, I neglected this. What is the value of
patience
? I think there may still be some logical errors for model saving.
@yzhangcs
I use default value of patience
, so it is 100. The patience
changes its value during training but I did not record it.
Ahhh, so I need to change value of patience
to 10 if I want more earlier stopping right?
def train(self, train, dev, test, buckets=32, batch_size=5000, update_steps=1,
clip=5.0, epochs=5000, patience=100, **kwargs):
...
@rootofmylife Yes, patience
controls the number of continued epochs after the best one.
@yzhangcs thanks for your information.
For the best metric problem, I have an updated discovery. I run a new training and see that the best value is probably evaluated on LAS. Please check my log (I also removed some redundant logs).
As you could see, the epoch 34 was saved because it had the highest score on LAS. So I checked the code and it was right (for the old and latest version). I guess your code is ok, nothing's wrong with the logic.
But I have another question, why don't we compare score using UAS or maybe both UAS and LAS? Moreover, we can save two models, one for the best LAS, one for the best UAS, can't we?
Thank you.
2021-09-16 05:19:43 INFO Epoch 25 / 100:
2021-09-16 05:20:36 INFO lr: 4.1872e-05 - loss: 0.0670 - UCM: 51.90% LCM: 26.82% UAS: 95.68% LAS: 90.48%
2021-09-16 05:20:37 INFO dev: loss: 1.2132 - UCM: 26.50% LCM: 13.00% UAS: 86.15% LAS: 78.73%
2021-09-16 05:20:39 INFO test: loss: 1.2733 - UCM: 26.96% LCM: 12.94% UAS: 84.86% LAS: 78.11%
2021-09-16 05:20:46 INFO 0:00:56.394561s elapsed (saved)
...
2021-09-16 05:22:39 INFO Epoch 28 / 100:
2021-09-16 05:23:32 INFO lr: 4.0230e-05 - loss: 0.0390 - UCM: 57.64% LCM: 30.00% UAS: 96.48% LAS: 91.69%
2021-09-16 05:23:33 INFO dev: loss: 1.3105 - UCM: 26.50% LCM: 11.50% UAS: 86.10% LAS: 78.73%
2021-09-16 05:23:35 INFO test: loss: 1.4364 - UCM: 27.06% LCM: 12.55% UAS: 85.01% LAS: 78.26%
2021-09-16 05:23:35 INFO 0:00:56.257415s elapsed
021-09-16 05:27:21 INFO Epoch 33 / 100:
2021-09-16 05:28:14 INFO lr: 3.7493e-05 - loss: 0.0374 - UCM: 65.09% LCM: 36.39% UAS: 97.49% LAS: 93.43%
2021-09-16 05:28:15 INFO dev: loss: 1.5302 - UCM: 27.00% LCM: 13.00% UAS: 86.06% LAS: 78.64%
2021-09-16 05:28:17 INFO test: loss: 1.6383 - UCM: 25.69% LCM: 12.84% UAS: 84.95% LAS: 78.03%
2021-09-16 05:28:17 INFO 0:00:56.611270s elapsed
2021-09-16 05:28:17 INFO Epoch 34 / 100:
2021-09-16 05:29:11 INFO lr: 3.6946e-05 - loss: 0.0414 - UCM: 66.41% LCM: 36.79% UAS: 97.64% LAS: 93.68%
2021-09-16 05:29:12 INFO dev: loss: 1.5116 - UCM: 28.00% LCM: 12.00% UAS: 86.01% LAS: 78.78%
2021-09-16 05:29:14 INFO test: loss: 1.6564 - UCM: 25.98% LCM: 12.94% UAS: 84.94% LAS: 78.16%
2021-09-16 05:29:20 INFO 0:00:56.750563s elapsed (saved)
2021-09-16 05:33:06 INFO Epoch 39 / 100:
2021-09-16 05:34:00 INFO lr: 3.4209e-05 - loss: 0.0026 - UCM: 72.46% LCM: 43.89% UAS: 98.22% LAS: 95.02%
2021-09-16 05:34:00 INFO dev: loss: 1.7252 - UCM: 27.00% LCM: 11.00% UAS: 86.01% LAS: 78.43%
2021-09-16 05:34:02 INFO test: loss: 1.8434 - UCM: 26.47% LCM: 12.84% UAS: 85.03% LAS: 78.08%
2021-09-16 05:34:02 INFO 0:00:56.636734s elapsed
2021-09-16 05:52:53 INFO Epoch 60 / 100:
2021-09-16 05:53:46 INFO lr: 2.2715e-05 - loss: 0.0057 - UCM: 86.14% LCM: 68.07% UAS: 99.22% LAS: 97.94%
2021-09-16 05:53:47 INFO dev: loss: 2.2656 - UCM: 28.50% LCM: 10.00% UAS: 86.13% LAS: 78.40%
2021-09-16 05:53:49 INFO test: loss: 2.4938 - UCM: 26.76% LCM: 13.04% UAS: 85.20% LAS: 78.32%
2021-09-16 05:53:49 INFO 0:00:56.310534s elapsed
...
2021-09-16 06:31:56 INFO Epoch 34 saved
2021-09-16 06:31:56 INFO dev: UCM: 28.00% LCM: 12.00% UAS: 86.01% LAS: 78.78%
2021-09-16 06:31:56 INFO test: UCM: 25.98% LCM: 12.94% UAS: 84.94% LAS: 78.16%
2021-09-16 06:31:56 INFO 1:34:07.910651s elapsed, 0:00:56.479107s/epoch
@rootofmylife
But I have another question, why don't we compare score using UAS or maybe both UAS and LAS? Moreover, we can save two models, one for the best LAS, one for the best UAS, can't we?
Yes, but you may need to implement a Metric
class yourself.
https://github.com/yzhangcs/parser/blob/191880d4a0417280e45c8d7595711b71c5427d61/supar/utils/metric.py#L59-L61
The score
property defines the values you wish to compare between different metrics.
@yzhangcs
Yeah, I got it, so I will close the issue here. Thank you for your support.
Have a good day.
Hello,
I wonder this is a bug or not. When I trained a model with 100 epochs, the last epoch had the highest metric but it did not save, instead of saving another lower metric model. See my log for more information.
Moreover, I remember that in old version, the parser had early stopping, but I did not see it in this version. It is really a cool features though.
Thank you for your help.
P/S: I use the latest code and not change anything.