search

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
1.98k stars 255 forks source link

Evaluation of empty strings with MAUVE results in error #593

Open lowlypalace opened 4 months ago

lowlypalace commented 4 months ago

Hi, I'm currently using HF wrapper to evaluate strings with MAUVE: https://huggingface.co/spaces/evaluate-metric/mauve

I'm getting the following an error when passing empty strings (e.g. ""). Is this an expected behavior? I would think that the empty string could be considered a valid string and therefore could be evaluated with MAUVE?

File "src/main.py", line 194, in <module>
    main()
  File "src/main.py", line 185, in main
    mauve_results_local, mauve_results_global = evaluate(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/global-decoding/src/eval/evaluate.py", line 58, in evaluate
    mauve_results_global = mauve.compute(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/evaluate/module.py", line 462, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/home/users/dgareev/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mauve/653f4690bcc6e9e16df42f9f85bd0f1b59cca1ae2864058dceedc595f26f6705/mauve.py", line 136, in _compute
    out = compute_mauve(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/compute_mauve.py", line 87, in compute_mauve
    p_features = get_features_from_input(
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/compute_mauve.py", line 176, in get_features_from_input
    features = featurize_tokens_from_model(MODEL, tokenized_texts, batch_size, name).detach().cpu().numpy()
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/mauve/utils.py", line 114, in featurize_tokens_from_model
    outs = model(input_ids=padded_chunk,
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/aiongpfs/projects/ai-art/global-decoding/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 777, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous