shaked571
commented
4 years ago Not to open to many threads I will continue here.
I belive that there is a bug in the current version. When you calculate the F1 score you in one hand adding "^" in the beginning of every POS, but in the other hand you don't strip them in the evaluation.
In my machine I changed the functions calculate_accuracy and existence_accuracy to
def calculate_accuracy(df):
numOfCorrectPredictions = 0
for index in df.index:
orig_pos = df.at[index, 'orig_label'].strip('^')
pred_pos = df.at[index, 'predicted_tag'].strip('^')
if orig_pos == pred_pos:
numOfCorrectPredictions += 1
return numOfCorrectPredictions/len(df)
and
def existence_accuracy(df):
# correct tag = appeared in predicted and in gold
total_orig_num_of_labels = 0
total_predicted_num_of_labels = 0
total_num_of_correct_tags = 0
for index in df.index:
orig_list = df.at[index, 'orig_label']
**predicted_list = df.at[index, 'predicted_tag'].strip('^')**
total_orig_num_of_labels += len(orig_list)
total_predicted_num_of_labels += len(predicted_list)
total_num_of_correct_tags += len(set(orig_list).intersection(set(predicted_list)))
precision = total_num_of_correct_tags / total_predicted_num_of_labels * 100
recall = total_num_of_correct_tags / total_orig_num_of_labels * 100
f1 = 2*precision*recall/(precision+recall)
print("Precision: {0:.2f}%".format(precision))
print("Recall: {0:.2f}%".format(recall))
print("F1: {0:.2f}%".format(f1))I would like to verfiy with you it is valid. Before my F1 score was 0, and now its better but still far from good: DEV: Precision: 55.47% Recall: 71.82% F1: 62.59% TEST: Precision: 53.74% Recall: 72.04% F1: 61.56%
stavkl
Under the add-line "Add sorting on the dataframes"
train_df['sent_len'] = train_df.groupby('sent_id').id.transform('size') s = train_df.sort_values(by=['sent_len', 'sent_id', 'id']).indexis raising an exception, I changed the lines to :train_df['sent_len'] = train_df.groupby('sent_id').sent_id.transform('size') s = train_df.sort_values(by=['sent_len', 'sent_id']).index