search

mark-xhchen / IE-CNN

IE-CNN model for ECPE task - COLING 2020 Paper
4 stars 0 forks source link

cant get the result #2

Closed tututu-Panda closed 3 years ago

tututu-Panda commented 3 years ago

I have followed what you said in "readme.MD", but I only got the result of f1 0.6148.

mark-xhchen commented 3 years ago

I have followed what you said in "readme.MD", but I only got the result of f1 0.6148.

Hi, I am rerunning the codes right now since it has been a year when I posted the codes. According to my previous logs, the average results should be 0.64~0.66 due to the effect of randomness.

I am wondering what code you have run to get 0.6148. Is it "main_cnn.py"? Is this result "cnn-softmax" or "cnn-crf"? Please let me know more details so I can locate the problems in my code. T T

tututu-Panda commented 3 years ago

I just runed the "cnn-crf". and I did not change the parameters in "main_cnn.py".

mark-xhchen commented 3 years ago

I just runed the "cnn-crf". and I did not change the parameters in "main_cnn.py".

I am still running the code, but as far as what I can observe, everything works normally for me.

Is 0.6148 the average F1 output for pair prediction, or is it the result of just a single fold? We have conducted the 10-fold cross-validation scheme, so the reported result is an average one, and based on my previous logs, this average value should be from 0.64-0.66. Can you provide a screenshot about the final summarized output of the cause, emotion, and pair prediction for me?

mark-xhchen commented 3 years ago

I just runed the "cnn-crf". and I did not change the parameters in "main_cnn.py".

something looks like this, should be in the last few lines of the output log file:

cause_predict: test f1 in 10 fold: [[0.66666667] [0.70072993] [0.67010309] [0.65116279] [0.68367347] [0.65104167] [0.65782493] [0.67156863] [0.59665871] [0.59259259]] average : acc 0.9526 p 0.7172 r 0.6031 f1 0.6542

position_predict: test f1 in 10 fold: [[0.82063882] [0.83950617] [0.80904523] [0.82352941] [0.82951654] [0.81329923] [0.84536082] [0.84261501] [0.77750611] [0.76756757]] average : acc 0.9746 p 0.8546 r 0.7831 f1 0.8169

position_predict: test f1 in 10 fold: [[0.67525723] [0.67294067] [0.64432939] [0.65415499] [0.69002645] [0.64673863] [0.68333283] [0.67331621] [0.595533 ] [0.64353262]] average : p 0.6939 r 0.6262 f1 0.6579

mark-xhchen commented 3 years ago

I just runed the "cnn-crf". and I did not change the parameters in "main_cnn.py".

Hi, I have finished rerunning the code, and the results I get for "cnn_crf" is as follows: image

If you haven't change anything about "main_cnn.py", it should produce two log files, one ending with "cnn_softmax.log" and the other one ending with "cnn_crf.log".

Have you finished running the code? It is possible that for a single fold the F1 value of pair recognition is low. But if you look at the average F1 across 10 folds, the F1 value should be around 0.64-0.66, or even higher.

tututu-Panda commented 3 years ago

I just rechecked this code and reran this code, I get results as follows:

cause_predict: test f1 in 10 fold: [[0.60732984] [0.60504202] [0.64150943] [0.6183844 ] [0.55172414] [0.61281337] [0.61741425] [0.61576355] [0.52691218] [0.59562842]] average : acc 0.9481 p 0.7062 r 0.5222 f1 0.5993

position_predict: test f1 in 10 fold: [[0.75510204] [0.78431373] [0.77806789] [0.75 ] [0.74680307] [0.73854447] [0.76092545] [0.78624079] [0.76719577] [0.77368421]] average : acc 0.9686 p 0.8430 r 0.7002 f1 0.7641

pair_predict: test f1 in 10 fold: [[0.60377308] [0.63742641] [0.64957215] [0.62209253] [0.564841 ] [0.64396235] [0.62146842] [0.5940589 ] [0.56534905] [0.60818664]] average : p 0.6859 r 0.5528 f1 0.6111

mark-xhchen commented 3 years ago

That's strange...

Can you check whether the "w2v_200.txt" under the folder "data_rand" is correct? If you directly download my code, this file should have no content, and hence the word embedding may all be randomly initialized. The correct file should have a size of around 80Mb.

tututu-Panda commented 3 years ago

Oh, sorry. I just noticed that this file in the remote host is just 4kb, I'll try it again

mark-xhchen commented 3 years ago

Oh, sorry. I just noticed that this file in the remote host is just 4kb, I'll try it again

Sure. I am also running the code without the pre-trained word embedding. I think this should be the main issue. See if you can get a reasonable performance this time.