huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
1.9k stars 235 forks source link

AttributeError: 'CombinedEvaluations' object has no attribute 'evaluation_modules' #603

Open shunk031 opened 3 weeks ago

shunk031 commented 3 weeks ago

Hi, I tried to run the following script by combining two metrics, but the error occurred:

Python 3.9.16 (main, Apr  5 2023, 13:58:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.18.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import evaluate

In [2]: evaluations = ("accuracy", "f1", "precision", "recall")

In [3]: clf_metrics = evaluate.combine(evaluations=evaluations)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[3], line 1
----> 1 clf_metrics = evaluate.combine(evaluations=evaluations)

File ~/.pyenv/versions/3.9.16/envs/my-dev/lib/python3.9/site-packages/evaluate/module.py:1034, in combine(evaluations, force_prefix)
   1008 def combine(evaluations, force_prefix=False):
   1009     """Combines several metrics, comparisons, or measurements into a single `CombinedEvaluations` object that
   1010     can be used like a single evaluation module.
   1011
   (...)
   1031     ```
   1032     """
-> 1034     return CombinedEvaluations(evaluations, force_prefix=force_prefix)

File ~/.pyenv/versions/3.9.16/envs/my-dev/lib/python3.9/site-packages/evaluate/module.py:884, in CombinedEvaluations.__init__(self, evaluation_modules, force_prefix)
    881     self.evaluation_module_names = list(evaluation_modules.keys())
    882 loaded_modules = []
--> 884 for module in self.evaluation_modules:
    885     if isinstance(module, str):
    886         module = load(module)

AttributeError: 'CombinedEvaluations' object has no attribute 'evaluation_modules'

If we use a list format for evaluations, it works fine, but I think we should also support tuple format, which work in the same way as list. This can be easily achieved by modifying the following section.

https://github.com/huggingface/evaluate/blob/main/src/evaluate/module.py#L877

May I create a PR?

pzdkn commented 3 weeks ago

@shunk031 did you?

shunk031 commented 3 weeks ago

Hi, I created the PR at #604.