Closed Asafgendler closed 1 year ago
That was on test set.
On val set: (1) word level evaluation
Model | Precision | Recall | F1 | Tightness | PQ |
---|---|---|---|---|---|
word-based, 128 queries | 0.7392418665 | 0.5715649016 | 0.6446788573 | 0.7857767433 | 0.5065736529 |
word-based, 256 queries | 0.7427441962 | 0.7035011985 | 0.7225902798 | 0.7822232082 | 0.5652268869 |
word-based, 384 queries | 0.7838868841 | 0.790175088 | 0.7870184257 | 0.7984368898 | 0.628384544 |
(2) para level eval | Model | Precision | Recall | F1 | Tightness | PQ |
---|---|---|---|---|---|---|
word-based, 128 queries | 0.6047776727 | 0.3567940167 | 0.448809085 | 0.7640780393 | 0.3429251657 | |
word-based, 256 queries | 0.6062317022 | 0.3827092345 | 0.4692099641 | 0.7615745829 | 0.3573383827 | |
word-based, 384 queries | 0.654185022 | 0.4061340105 | 0.5011449922 | 0.783455846 | 0.3926249738 |
No line level eval here. It just doesn't make any sense.
Likewise, on the test set:
(1) word level eval
Model | Precision | Recall | F1 | Tightness | PQ |
---|---|---|---|---|---|
128 queries | 0.750084691 | 0.5849323951 | 0.6572932128 | 0.7874197285 | 0.5175656431 |
256 queries | 0.7518331545 | 0.7153329811 | 0.733129042 | 0.781384374 | 0.5728555776 |
384 queries | 0.793973056 | 0.8067676937 | 0.8003192414 | 0.7983057275 | 0.6388994342 |
(2) para level eval | Model | Precision | Recall | F1 | Tightness | PQ |
---|---|---|---|---|---|---|
128 queries | 0.6167621031 | 0.3659274816 | 0.4593316275 | 0.7693506585 | 0.3533870901 | |
256 queries | 0.6216555308 | 0.3744629069 | 0.4673880702 | 0.7598261086 | 0.3551336586 | |
384 queries | 0.6593808676 | 0.4230032738 | 0.5153812866 | 0.782588586 | 0.4033315123 |
Hello, and thanks for the great dataset,
Will it be possible to publish the results of the word-based model for the line level evaluation? (validation set) also the results you published in a previous issue using the word-based model for words level detection are for the validation or test set? if they are for the test can you show it for the validation set also? and also the results for the paragraphs level evaluation on the validation set would be appreciated as the paper only shows them on the test set.