search

kasnerz / factgenie

A Toolkit for Annotating and Visualizing LLM Hallucinations
MIT License
1 stars 0 forks source link

User-friendly configuration for LLM eval #5

Open kasnerz opened 3 weeks ago

kasnerz commented 3 weeks ago

The configuration of LLM eval currently sits only within YAML files in factgenie/llm-eval. The files contains parameters such as prompt, system message, or decoding parameters.

When creating a new LLM eval, the user needs to select one of these files (according to its filename).

In the very least, we need to document how to create these files and preview their content during the selection process.

In the long run, it would be better to make the individual fields editable from the web interface.

screen-2024-06-10-14-31-13

oplatek commented 5 days ago

I would actually prefer that the UI is able to load the configuration file, edit selected fields, and save them.

In ideal case, it would not matter if you edit them locally or through the web browser UI.

oplatek commented 4 days ago

@kasnerz note that in #32 I added the errors_categories as a mandatory field in the config next to the prompt because they need to match anyway.

I did not change the frontend UI, but I think it makes sense to save the whole configuration in the single factgenie/llm-eval/MY_LLM_METRIC_CONFIG.YAML including the annotations_categories (errors_categories), prompt and make the config editable in the browser UI. (As I mentioned also above).