[Question] After training, how to use trained model?

[parksjin01]

Hi, I'm student who studies NLP.

I trying to use dep2label model for dependency parsing task and I have some questions about result.

First, UAS of dev dataset is too high. When using Relative PoS-based encoding, UAS shows about 99% with 200 iterations. I think it's too high even if it is dev dataset

Second, after training the model, when I use trained model to check test dataset accuracy
UAS shows under 40%.
I can't infer the reason why that kinds of things happen.

Third, in main.py line 486 encoding is fixed as 0 but in line 498 decoding is set as 3.
I have a question that if encoding and decoding method is different, it doesn't affect to performance?

Below is the config file for training and decode.

train.config

### use # to comment out the configure item

### I/O ###
train_gold=nkor/train.conll
dev_gold=nkor/test.conll
# test_gold=nkor/test.conll
train_enc_dep2label=myModel/encodedDataTrain.seq
dev_enc_dep2label=myModel/encodedDataDev.seq
# test_enc_dep2label=myModel/encodedDatatest.seq
model_dir=myModel/dep2label
word_emb_dir=embedding/embedding.txt
encoding=3
eval_type=CONLL
postag_type=UPOS

norm_word_emb=True
norm_char_emb=False
number_normalized=False
seg=False
word_emb_dim=100
char_emb_dim=50

###NetworkConfiguration###
use_crf=False
use_char=True
word_seq_feature=LSTM
char_seq_feature=CNN
feature=[POStag] emb_size=50

###TrainingSetting###
status=train
optimizer=adadelta
iteration=200
batch_size=256
ave_batch_loss=True

###Hyperparameters###
cnn_layer=4
char_hidden_dim=100
hidden_dim=400
dropout=0.5
lstm_layer=2
bilstm=True
learning_rate= 0.02
lr_decay=0.05
momentum=0.9
l2=0
gpu=True
#clip=

decode.config

### Decode ###
status=decode
test_gold=nkor/test.conll
input=nkor/test.conll
raw_dir=myModel/encodedDataTest.seq
output_nn=myModel/test_output_nn.out
parsedTree=myModel/parsed.conll
dset_dir=myModel/dep2label.dset
load_model_dir=myModel/dep2label.model
gpu=True

decode_best_model.config

### Decode ###
status=decode
test_gold=nkor/dev_backup.conll
input=nkor/dev.conll
raw_dir=myModel/encodedDataDev.seq
output_nn=myModel/output_nn.out
parsedTree=myModel/parsed.conll
dset_dir=myModel/dep2label.dset
load_model_dir=myModel/dep2label.model
encoding=3
gpu=True

File format

• nkor/train.conll
• nkor/dev_backup.conll
• nkor/test.conll

• above 3 file contains arc, label information

• nkor/dev.conll
• above 1 file doesn't contain arc, label information

[mstrise]

Hi! Thanks for your interest in Sequence Labeling Parsing.

First of all, you should only use training and development set during training, never test set, since the latter should never be shown and be evaluated on during training (I'm referring to this line dev_gold=nkor/test.conll). Maybe if you change it to the file with dev set it will give you the correct accuracy.

In line 486 all labels from the file you want to evaluate your model on, are set to 0. The point of that is to assure that a model is predicting those labels.