search

microsoft / vert-papers

This repository contains code and datasets related to entity/knowledge papers from the VERT (Versatile Entity Recognition & disambiguation Toolkit) project, by the Knowledge Computing group at Microsoft Research Asia (MSRA).
MIT License
266 stars 93 forks source link

Why my results are so poor? #89

Open sehbe opened 10 months ago

sehbe commented 10 months ago

I downloaded the code without changing anything, just to make sure it ran, I loaded the local bert_model

2023-10-26 12:59:12 INFO: - Step: 0/5001, span loss = 6.118810, type loss = 0.000000, time = 2.27s. 2023-10-26 12:59:53 INFO: - Step: 20/5001, span loss = 5.980084, type loss = 0.000000, time = 43.38s. 2023-10-26 13:00:36 INFO: - Step: 40/5001, span loss = 5.879471, type loss = 0.000000, time = 86.74s. 2023-10-26 13:01:17 INFO: - Step: 60/5001, span loss = 5.772162, type loss = 0.000000, time = 128.07s. 2023-10-26 13:01:57 INFO: - Step: 80/5001, span loss = 5.848635, type loss = 0.000000, time = 167.65s. 2023-10-26 13:02:36 INFO: - Step: 100/5001, span loss = 5.735874, type loss = 0.000000, time = 206.93s. 2023-10-26 13:03:15 INFO: - Step: 120/5001, span loss = 5.667262, type loss = 0.000000, time = 246.25s. 2023-10-26 13:03:55 INFO: - Step: 140/5001, span loss = 5.633423, type loss = 0.000000, time = 285.52s. 2023-10-26 13:04:34 INFO: - Step: 160/5001, span loss = 5.501272, type loss = 0.000000, time = 325.18s. 2023-10-26 13:05:16 INFO: - Step: 180/5001, span loss = 5.545497, type loss = 0.000000, time = 366.45s. 2023-10-26 13:05:55 INFO: - Step: 200/5001, span loss = 5.426091, type loss = 0.000000, time = 406.06s.

2023-10-26 13:08:26 INFO: - Eval results inter-valid 2023-10-26 13:08:26 INFO: - f1 = 0.0 2023-10-26 13:08:26 INFO: - f1_threshold = 0.0 2023-10-26 13:08:26 INFO: - loss = tensor(1.1085, device='cuda:0') 2023-10-26 13:08:26 INFO: - precision = 0.0 2023-10-26 13:08:26 INFO: - precision_threshold = 0.0 2023-10-26 13:08:26 INFO: - recall = 0.0 2023-10-26 13:08:26 INFO: - recall_threshold = 0.0 2023-10-26 13:08:26 INFO: - span_f1 = 0.05210224202151717 2023-10-26 13:08:26 INFO: - span_p = 0.03787716670233228 2023-10-26 13:08:26 INFO: - span_r = 0.08343808925204142 2023-10-26 13:08:26 INFO: - type_f1 = 0.0 2023-10-26 13:08:26 INFO: - type_p = 0.0 2023-10-26 13:08:26 INFO: - type_r = 0.0 2023-10-26 13:08:26 INFO: - 0.000,0.000,0.000,3.788,8.344,5.210,0.000,0.000,0.000,0.000,0.000,0.000 2023-10-26 13:08:26 INFO: - ===> Best Valid F1: 0.0

iofu728 commented 10 months ago

Hi @sehbe, according to the logs, it seems that you haven't trained for a sufficient amount of time. DecomposedMetaNER is a two-stage process that requires separate training for the Span Detector and Entity Typing. Our method is not zero-shot and needs to be trained on a few-shot dataset.

Additionally, it's normal for the type and final F1 to be 0 during the Span Detector stage. You'll need to wait until both classifiers are fully trained to see the final results. As also mentioned in #66.