DeNederlandscheBank / nqm

A Transformer-based Machine for answering questions on insurance companies

MIT License

0 stars 0 forks source link

evaluation not correctly working #38

Closed jm-glowienke closed 3 years ago

jm-glowienke commented 3 years ago

For subword processing

Reason:

The input to fairseq-interactive is already BPE processed, so it gets double BPE processed, causing bad perfomance

Resolve by:

[x] copy extra files to the model_input folder, which are not BPE processed when evaluating
[x] change input file in relevant script evaluation alignment

General:

Traceback (most recent call last):
  File "src_eiopa/query_results_evaluation.py", line 101, in <module>
    acc, results = compare_results(g, queries)
  File "src_eiopa/query_results_evaluation.py", line 41, in compare_results
    if result_true[index] == result_generated[index]:

[x] ~~use zip formulation~~ use for loop for reference results and in formulation for checking