Bug: Cannot export evaluation report when responses contain failed results - Githubissues

UBC-MDS / fixml

LLM Tool for effective test evaluation of ML projects with curated Checklists and LLM prompts

https://ubc-mds.github.io/fixml

Other

3 stars 2 forks source link

Bug: Cannot export evaluation report when responses contain failed results #190

Closed tonyshumlh closed 4 months ago

tonyshumlh commented 4 months ago

Descriptions:

Cannot export evaluation report when responses contain failed results

How to reproduce the bug:

Run the command fixml evaluate ../magenta -e eval_report.qmd -v -o where magenta is from https://github.com/magenta/magenta
The error message:

Inspection:

probably because of the failed results handling in the code in parse.py
can produce evaluation report after workaround
please note the impact to the content in the evaluation report and edit if needed