search

patverga / bran

Full abstract relation extraction from biological texts with bi-affine relation attention networks
Apache License 2.0
128 stars 33 forks source link

Questions regarding CDR results and noise_classifier #3

Closed arwhirang closed 6 years ago

arwhirang commented 6 years ago

Hello.

I have run this bran code for the CDR dataset several times and now I have few questions.

I found that the results printed on the STDOUT are from the ner_eval and the relation_eval functions. Also I found that the "Best" score in the following sentence "Eval decreased for %d epochs out of %d max epochs. Best: %2.2f" is the result from the development part most of the time.

For some reason, when I run the code for CDR dataset (with no additional data), the F-score from the test set is 60% in average. I think my configuration is the problem. I have following weird STDOUT logs. But I got stuck here with no improvement for days. Could you give me some help, please?

Couldnt get noise_classifier/token_embeddings Couldnt get noise_classifier/w_1 Couldnt get noise_classifier/b_1 Couldnt get noise_classifier/pos_encoding Couldnt get noise_classifier/nuW Couldnt get noise_classifier/num_blocks_0/multihead_attention/dense/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention/dense_1/kernel Couldnt get noise_classifier/num_blocks_0/multihead_attention/dense_1/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention/dense_2/kernel Couldnt get noise_classifier/num_blocks_0/multihead_attention/dense_2/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention/ln/Variable Couldnt get noise_classifier/num_blocks_0/multihead_attention/ln/Variable_1 Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d/kernel Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d_1/kernel Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d_1/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d_2/kernel Couldnt get noise_classifier/num_blocks_0/multihead_attention/conv1d_2/bias Couldnt get noise_classifier/num_blocks_0/multihead_attention_1/ln/Variable Couldnt get noise_classifier/num_blocks_0/multihead_attention_1/ln/Variable_1 Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense_1/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense_1/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense_2/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/dense_2/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention/ln/Variable Couldnt get noise_classifier/num_blocks_1/multihead_attention/ln/Variable_1 Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d_1/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d_1/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d_2/kernel Couldnt get noise_classifier/num_blocks_1/multihead_attention/conv1d_2/bias Couldnt get noise_classifier/num_blocks_1/multihead_attention_1/ln/Variable Couldnt get noise_classifier/num_blocks_1/multihead_attention_1/ln/Variable_1 Couldnt get noise_classifier/text/dense/kernel Couldnt get noise_classifier/text/dense/bias Couldnt get noise_classifier/text/dense_1/kernel Couldnt get noise_classifier/text/dense_1/bias Couldnt get noise_classifier/text/dense_2/kernel Couldnt get noise_classifier/text/dense_2/bias Couldnt get noise_classifier/text/dense_3/kernel Couldnt get noise_classifier/text/dense_3/bias Couldnt get noise_classifier/text/Bilinear/Weights Couldnt get noise_classifier/num_blocks_0_1/multihead_attention/ln/Variable Couldnt get noise_classifier/num_blocks_0_1/multihead_attention/ln/Variable_1 Couldnt get noise_classifier/num_blocks_0_1/multihead_attention_1/ln/Variable Couldnt get noise_classifier/num_blocks_0_1/multihead_attention_1/ln/Variable_1 Couldnt get noise_classifier/num_blocks_1_1/multihead_attention/ln/Variable Couldnt get noise_classifier/num_blocks_1_1/multihead_attention/ln/Variable_1 Couldnt get noise_classifier/num_blocks_1_1/multihead_attention_1/ln/Variable Couldnt get noise_classifier/num_blocks_1_1/multihead_attention_1/ln/Variable_1 Couldnt get noise_classifier/score_sentence/ner_w_1 Couldnt get noise_classifier/score_sentence/ner_b_1

By the way, thanks for the great work.

arwhirang commented 6 years ago

The performance difference may be due to the embedding dimension? In the paper, the authors used embedding dimension 128 for CTD dataset, while default code setting of dimension size is 64. Then again, why do you set the default dimension size as 64?

patverga commented 6 years ago

Hi,

I don't think it is the embedding dimension, the numbers I got were using 64 dim for the CDR dataset. Running the code as is will train a model with early stopping defined by the 'best score' on the development set. Then that best model would be used to generate a test set score.

A few possibilities for your issue could be 1- the train / development split is being selected by a random 85/15 % split. This causes variance in the performance of the model. The reported numbers were averaged over 10 runs. 2- the tensorflow version could be impacting the results. I was using tf version 1.0.1. I noticed different versions of tf sometimes led to slight variations in the models performance.

arwhirang commented 6 years ago

Thank you for the answer. I am using the tensorflow 1.0.1 version. The train/dev split may be the cause, I guess?

What do you think about the "Couldnt get noise_classifier/score_sentence/ner_w_1" log? If you give a hint, I could look for the root of the problem.

patverga commented 6 years ago

This message is coming from here: https://github.com/patverga/bran/blob/master/src/train_multiclass_classifier.py#L362

This is a vague and poorly worded warning that comes up when creating moving averages for the trainable variables in the graph. This is something I need to fix.

For each variable in tf.trainable_variables, we try to set up a moving average. For those that do not succeed, it prints out this 'could not get variable' message. I believe that this may be something you don't have to worry about and results from tf.trainable_variables simply returning things that can't be set up with a moving average.

arwhirang commented 6 years ago

Okay, Thanks for a lot of help.