关于ontonote,weibo和resume数据集上实验结果问题

[gloria0108]

作者您好： 我用ontonote,weibo,resume,msr数据集（BMEOS模式）在您的原始代码（参数等均未改动）上跑了使用average strategy的实验，其中在msr数据集上能完全复现您论文汇报的结果。其余数据集上比您论文汇报的结果要低一些，请问是由于随机种子的原因还是需要额外的调参？ 我跑出的实验结果如下： ontonote:73.36（论文73.98） weibo nm:63.76（论文64.17） weibo all:57.52（论文58.67） resume:94.21（论文95.03） 对应的log如下。谢谢！

log.train.note4.txt log.train.resume.txt log.train.weibo.nm.txt log.train.weibo.all.txt

[liuwei1206]

hi，您好

可能是因为随机种子的原因，其他超参数应该不用调；事实上，我的超参数大部分是参考Lattice LSTM的，也没有经过微调。

不过，您说效果偏低，我表示很疑惑，因为后面代码是我跑过之后传上去的，效果大部分比论文还要好。但我是分开跑的，不知道是不是整合的时候出现了一些问题。

我看了你发的日志文件，Ontonote4里面差别挺大的，我将Ontonote4的average策略的代码上传到云盘，你可以再试一试！ 链接：https://pan.baidu.com/s/1CXnUwvtPbkjMDUIIWNNU3w 提取码：gv2y

希望对比有帮助！

[liuwei1206]

另外，我还从github上直接clone下来代码，跑了一下weibo的实验，用average策略，我只跑了23大的epoch，效果就已经很明显了，下面是结果的截图： Epoch: 19 training finished. Time: 139.70s, speed: 9.66st/s, total loss: 2160.899757385254 gold_num = 389 pred_num = 364 right_num = 235 Dev: time: 2.02s, speed: 134.20st/s; acc: 0.9595, p: 0.6456, r: 0.6041, f: 0.6242 Exceed previous best f score: 0.6231884057971016 gold_num = 418 pred_num = 357 right_num = 232 Test: time: 1.84s, speed: 147.24st/s; acc: 0.9558, p: 0.6499, r: 0.5550, f: 0.5987 Epoch: 20/50 Learning rate is setted as: 0.005377288836128128 Instance: 1350; Time: 142.40s; loss: 2099.8514; acc: 72292.0/73778.0=0.9799 Epoch: 20 training finished. Time: 142.40s, speed: 9.48st/s, total loss: 2099.851402282715 gold_num = 389 pred_num = 347 right_num = 221 Dev: time: 2.09s, speed: 129.88st/s; acc: 0.9586, p: 0.6369, r: 0.5681, f: 0.6005 gold_num = 418 pred_num = 330 right_num = 218 Test: time: 2.21s, speed: 122.77st/s; acc: 0.9549, p: 0.6606, r: 0.5215, f: 0.5829 Epoch: 21/50 Learning rate is setted as: 0.005108424394321722 Instance: 1350; Time: 139.88s; loss: 2050.2969; acc: 72306.0/73778.0=0.9800 Epoch: 21 training finished. Time: 139.88s, speed: 9.65st/s, total loss: 2050.2968673706055 gold_num = 389 pred_num = 339 right_num = 228 Dev: time: 2.05s, speed: 132.50st/s; acc: 0.9604, p: 0.6726, r: 0.5861, f: 0.6264 Exceed previous best f score: 0.6241699867197875 gold_num = 418 pred_num = 337 right_num = 224 Test: time: 1.85s, speed: 146.41st/s; acc: 0.9568, p: 0.6647, r: 0.5359, f: 0.5934 Epoch: 22/50 Learning rate is setted as: 0.004853003174605635 Instance: 1350; Time: 141.37s; loss: 2057.7770; acc: 72323.0/73778.0=0.9803 Epoch: 22 training finished. Time: 141.37s, speed: 9.55st/s, total loss: 2057.776954650879 gold_num = 389 pred_num = 347 right_num = 233 Dev: time: 2.14s, speed: 126.53st/s; acc: 0.9603, p: 0.6715, r: 0.5990, f: 0.6332 Exceed previous best f score: 0.6263736263736264 gold_num = 418 pred_num = 329 right_num = 225 Test: time: 2.04s, speed: 132.74st/s; acc: 0.9565, p: 0.6839, r: 0.5383, f: 0.6024 Epoch: 23/50 Learning rate is setted as: 0.004610353015875353 Instance: 1350; Time: 141.84s; loss: 1966.1277; acc: 72389.0/73778.0=0.9812 Epoch: 23 training finished. Time: 141.84s, speed: 9.52st/s, total loss: 1966.1277313232422 gold_num = 389 pred_num = 327 right_num = 218 Dev: time: 2.13s, speed: 127.30st/s; acc: 0.9596, p: 0.6667, r: 0.5604, f: 0.6089 gold_num = 418 pred_num = 314 right_num = 220 Test: time: 1.93s, speed: 140.35st/s; acc: 0.9565, p: 0.7006, r: 0.5263, f: 0.6011

[gloria0108]

好的，我再试一下，非常感谢您的耐心回复！ 另外，请问您用github上clone下来的代码跑的这个weibo的实验是哪个数据集？all,nm还是ne？

[liuwei1206]

all的数据集

[gloria0108]

您好，抱歉再次打扰您，我在weibo.all数据集上还是没能重现出您论文汇报的结果。我对比了我的log日志和您的日志，发现我test数据集和您test数据集的gold_num不同，因此怀疑数据集存在差异。下面是我拿到的weibo.all数据集，请问是否方便分享一下您使用的weibo all数据集，以对比一下是否有哪些不同。非常感谢！ train.all.bmes.txt dev.all.bmes.txt test.all.bmes.txt

[liuwei1206]

我的weibo数据集已经传到github了呀

[gloria0108]

哦哦好的。抱歉，之前没注意。。。