-
The configuration of LLM eval currently sits only within YAML files in [factgenie/llm-eval](https://github.com/kasnerz/factgenie/tree/main/factgenie/llm-eval). The files contains parameters such as pr…
-
max_num_samples=-1で評価スクリプトを走らせていたところ、wikicorpus-e-to-jタスクでの評価終了後(間際?)にエラーが出て中断してしまいました。
データ処理などはReadmeに記載されている通りに実行しました。
また、max_num_samples=100でエラーなく完了できていたことを確認しています。
エラーの表記から察するに、BLEUでのcorpusレベルの…
-
## What is the problem?
If you change the model's name, the llm-eval ID, as listed in http://10.10.24.15:5000/llm_eval, will also change because it is created based on the config content, including…
-
I tried to run eval, but suffered some issues:
ModuleNotFoundError: No module named 'accessory.model.LLM.pointbert'
ModuleNotFoundError: No module named 'accessory.model.LLM.llama_qformerv2'
…
-
https://github.com/llm-jp/llm-jp-eval/pull/115 の `offline_inference_example.py` を参考にFastGenでオフライン推論処理を実装する。
-
G-Eval includes "Auto Chain-of-Thoughts for NLG Evaluation" as a component where the CoT steps to carry out evaluation are produced by an LLM. The paper nor this repo, however, include the prompt defi…
-
### Prerequisite
- [X] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expe…
-
[ ] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.
**Your Question**
> “WARNING:ragas.llms.output_parser:Failed to parse …
-
As a user, I would like to be informed about the summarization effectiveness of my chosen LLM endpoint.
I would like to be able to evaluate an endpoint against a known, tested framework, to evaluat…
-
When I run `evaluate` with any model of VertexAI, I get several warnings that say
> Gapic client context issue detected.This can occur due to parallelization.
And sometimes the execution of eva…