How to add a metric to parse from HELM output in wandb_writer.py?

[Mr-lonely0]

Before Asking 在提问之前

• [X] I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

• [X] I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

• [X] I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

I want to add a metric named Denoised inference time (s) to track the efficiency of my model in tools/evaluator/recorder/wandb_writer.py, so I changed the config file mymodel_example.yaml as follows:

project: ee_tune_eval
base_url: https://wandb.ai/liukui/EE-TUNE-EVAL
evals:
  - eval_type: helm
    model_name: llama2_ee
    source: helm
    helm_output_dir: /data3/lk/data-juicer/helm_output
    helm_suite_name: ee_test-llama2_ee
    token_per_iteration: <tokens per iteration in billions>
    benchmarks:
      - name: summarization_cnndm
        metrics:
          - ROUGE-2
          - Denoised inference time (s)
      - name: summarization_xsum
        metrics:
          - ROUGE-2
      - name: narrative_qa
        metrics:
          - F1
      - name: mmlu
        metrics:
          - EM

However, I got an error:

parsing summarization_cnndm.json
  parsing dataset_name: cnn-dm, sampling_min_length: 50, sampling_max_length: 150, doc_max_length: 512
Fail to parse summarization_cnndm: 'value'

I then checked the summarization_cnndm.json in HELM output dir, and there indeed isn't a value for Denoised inference time (s). What modifications should I make to parse Denoised inference time (s) from the HELM output?

I would greatly appreciate your help and look forward to your prompt response.

Additional 额外信息

No response

[pan-x-c]

Please refer to link

[github-actions[bot]]

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.