由于流量过大,本站暂不支持访问 Spaces 空间,可前往 Hugging Face 官网查看。
Open simplelifetime opened 5 months ago
Have you solved this problem? I also encountered
Same here
Same here
Same here with version 0.4.2
Same problem
Hey guys, I guess it's a problem with evaluate==0.4.2. I replaced the cached exact_match.py script (the file where the error occured) with a previous version (sry not sure which version it is exactly), and everything works fine from my side. Please feel free to use this as a temperory solution, but I think some bug need to be solved within version 0.4.2.
The exact_match.py I used: (I obtained it from my partner, credit to him :)
# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Exact Match metric."""
import re
import string
import datasets
import numpy as np
import evaluate
_DESCRIPTION = """
Returns the rate at which the input predicted strings exactly match their references, ignoring any strings input as part of the regexes_to_ignore list.
"""
_KWARGS_DESCRIPTION = """
Args:
predictions: List of predicted texts.
references: List of reference texts.
regexes_to_ignore: List, defaults to None. Regex expressions of characters to
ignore when calculating the exact matches. Note: these regexes are removed
from the input data before the changes based on the options below (e.g. ignore_case,
ignore_punctuation, ignore_numbers) are applied.
ignore_case: Boolean, defaults to False. If true, turns everything
to lowercase so that capitalization differences are ignored.
ignore_punctuation: Boolean, defaults to False. If true, removes all punctuation before
comparing predictions and references.
ignore_numbers: Boolean, defaults to False. If true, removes all punctuation before
comparing predictions and references.
Returns:
exact_match: Dictionary containing exact_match rate. Possible values are between 0.0 and 1.0, inclusive.
Examples:
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds)
>>> print(round(results["exact_match"], 2))
0.25
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell"], ignore_case=True, ignore_punctuation=True)
>>> print(round(results["exact_match"], 2))
0.5
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell", "YELL"], ignore_case=True, ignore_punctuation=True)
>>> print(round(results["exact_match"], 2))
0.75
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["the cat", "theater", "YELLING", "agent007"]
>>> preds = ["cat?", "theater", "yelling", "agent"]
>>> results = exact_match.compute(references=refs, predictions=preds, regexes_to_ignore=["the ", "yell", "YELL"], ignore_case=True, ignore_punctuation=True, ignore_numbers=True)
>>> print(round(results["exact_match"], 2))
1.0
>>> exact_match = evaluate.load("exact_match")
>>> refs = ["The cat sat on the mat.", "Theaters are great.", "It's like comparing oranges and apples."]
>>> preds = ["The cat sat on the mat?", "Theaters are great.", "It's like comparing apples and oranges."]
>>> results = exact_match.compute(references=refs, predictions=preds)
>>> print(round(results["exact_match"], 2))
0.33
"""
_CITATION = """
"""
@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
class ExactMatch(evaluate.Metric):
def _info(self):
return evaluate.MetricInfo(
description=_DESCRIPTION,
citation=_CITATION,
inputs_description=_KWARGS_DESCRIPTION,
features=datasets.Features(
{
"predictions": datasets.Value("string", id="sequence"),
"references": datasets.Value("string", id="sequence"),
}
),
reference_urls=[],
)
def _compute(
self,
predictions,
references,
regexes_to_ignore=None,
ignore_case=False,
ignore_punctuation=False,
ignore_numbers=False,
):
if regexes_to_ignore is not None:
for s in regexes_to_ignore:
predictions = np.array([re.sub(s, "", x) for x in predictions])
references = np.array([re.sub(s, "", x) for x in references])
else:
predictions = np.asarray(predictions)
references = np.asarray(references)
if ignore_case:
predictions = np.char.lower(predictions)
references = np.char.lower(references)
if ignore_punctuation:
repl_table = string.punctuation.maketrans("", "", string.punctuation)
predictions = np.char.translate(predictions, table=repl_table)
references = np.char.translate(references, table=repl_table)
if ignore_numbers:
repl_table = string.digits.maketrans("", "", string.digits)
predictions = np.char.translate(predictions, table=repl_table)
references = np.char.translate(references, table=repl_table)
score_list = predictions == references
return {"exact_match": np.mean(score_list)}
Same problem. I tried the solution proposed by @ErikaaWang and found it worked.
I can see this in blue.py
由于流量过大,本站暂不支持访问 Spaces 空间,可前往 Hugging Face 官网查看。
I guess the problem may be from the https://hf-mirror.com
I fixed the issue by changing HF_ENDPOINT to https://huggingface.co/.
Same problem, I tried the solution from @ZeguanXiaoand, but it didn't work out
Same problem, I tried the solution from @ZeguanXiaoand, but it didn't work out
Try: export HF_ENDPOINT="https://huggingface.co"
@sxluo Thanks for your advice! I originally have this line:
export HF_ENDPOINT=https://hf-mirror.com
However I cannot remove this line, or change this to https://huggingface.co
. Otherwise i wouldn't be able to connect to HF on my server. (mainland China).
So I clone 'evaluate' to my local path, and change this line:
metric = evaluate.load("accuracy")
to:
metric = evaluate.load(MY_PATH+"/evaluate/accuracy/accuracy.py")
It turns out to work well :>. Hope this could help others.
Here is my running code:
import evaluate ev = evaluate.load('exact_match')
Always return below error Downloading builder script: 3.69kB [00:00, 4.56MB/s] Traceback (most recent call last): File "", line 1, in
File "/home/aka/anaconda3/lib/python3.10/site-packages/evaluate/loading.py", line 751, in load
evaluation_cls = import_main_class(evaluation_module.module_path)
File "/home/aka/anaconda3/lib/python3.10/site-packages/evaluate/loading.py", line 76, in import_main_class
module = importlib.import_module(module_path)
File "/home/aka/anaconda3/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 879, in exec_module
File "", line 1017, in get_code
File "", line 947, in source_to_code
File "", line 241, in _call_with_frames_removed
File "/home/aka/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--exact_match/009c8b5313309ea5b135d526433d5ee76508ba1554cbe88310a30f85bb57ec88/exact_match.py", line 16
}
^
SyntaxError: closing parenthesis '}' does not match opening parenthesis '(' on line 14