search

renll / StateNet

[EMNLP 2018] Towards Universal Dialogue State Tracking
Other
42 stars 11 forks source link

Reproducing the results for DSTC2 #2

Closed kolk closed 5 years ago

kolk commented 5 years ago

Hi, I am unsure how to reproduce the results reported on the paper. I have generated the embeddings with get_embedding.py and the train_nbest.json and test_nbest.json with mat_data.py. To train, I am running the code offline_model_dstc.py but the accuracy is not improving. Am I missing something?

=============BEGIN EPOCH 0===================
('customAcc', 0.3751827485380117)
[Training over]  2019-06-24 17-08
train: [('customAcc', 0.0)]
dev  : [('customAcc', 0.0)]
test : [('customAcc', 0.0)]
[Testing over]  2019-06-24 17-11
devbest epoch: -1, acc: 0.0
testbest epoch: -1, acc: 0.0
=============BEGIN EPOCH 1===================
('customAcc', 0.32132292781169364)
[Training over]  2019-06-24 17-12
train: [('customAcc', 0.0)]
dev  : [('customAcc', 0.0)]
test : [('customAcc', 0.0)]
[Testing over]  2019-06-24 17-15
devbest epoch: -1, acc: 0.0
testbest epoch: -1, acc: 0.0
=============BEGIN EPOCH 2===================
('customAcc', 0.32132292781169364)
[Training over]  2019-06-24 17-17
train: [('customAcc', 0.0)]
dev  : [('customAcc', 0.0)]
test : [('customAcc', 0.0)]
[Testing over]  2019-06-24 17-20
devbest epoch: -1, acc: 0.0
testbest epoch: -1, acc: 0.0
=============BEGIN EPOCH 3===================
('customAcc', 0.32132292781169364)
[Training over]  2019-06-24 17-22
train: [('customAcc', 0.0)]
dev  : [('customAcc', 0.0)]
test : [('customAcc', 0.0)]
[Testing over]  2019-06-24 17-24
devbest epoch: -1, acc: 0.0
testbest epoch: -1, acc: 0.0
renll commented 5 years ago

You can have a try to switch the variable "ini" in the offline_model_dstc.py to 1 to the pretraining mode.

wangjinjie722 commented 4 years ago

Hi there,

I got the same issue, and below is my log:

====== Train labelIdx-99: [Start] 2020-08-04 20-21 =============BEGIN EPOCH 0=================== ('customAcc', 0.3751370614035088) [Training over] 2020-08-04 20-23 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-24 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 1=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-24 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-25 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 2=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-25 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-26 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 3=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-26 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-27 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 4=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-27 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-28 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 5=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-28 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-29 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 6=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-29 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-30 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 7=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-30 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-31 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 8=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-31 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-32 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 9=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-32 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-33 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 10=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-33 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-34 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 11=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-34 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-35 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 12=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-35 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-36 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 13=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-36 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-37 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247 =============BEGIN EPOCH 14=================== ('customAcc', 0.3968708852916118) [Training over] 2020-08-04 20-37 train: [('customAcc', 0.00214096086323542)] dev : [('customAcc', 0.003050330452465684)] test : [('customAcc', 0.003050330452465684)] [Testing over] 2020-08-04 20-37 devbest epoch: 0, acc: 0.00305033045247 testbest epoch: 0, acc: 0.00305033045247