Domain specific metrics - Githubissues

dfsbora / latex-math-model

0 stars 0 forks source link

Domain specific metrics #18

Open dfsbora opened 4 weeks ago

dfsbora commented 4 weeks ago

Evaluate valid formula, check for labels/refs/formulas in original text, etc

dfsbora commented 3 weeks ago

EvaluatePrompt class' run_all method (in utils.py) returns a dictionary with the metrics:

'valid_formulas': Evaluates the validity of LaTeX formulas.
'original_formulas': Compares the formulas with data formulas
'correct_refs_irt_prompt': Checks the references with labels in prompt.
'correct_refs_irt_data': Checks the references with labels in original data.
'original_cites': Compares the citations with data citations.
'original_labels': Compares the labels with data labels.

Still want to improve the formula comparison to account for slight modifications, rather than a binary check.