search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.55k stars 186 forks source link

Another metric with the same name already exists. #52

Closed pritamdeka closed 2 years ago

pritamdeka commented 2 years ago

Hi @NThakur20

There seems to be an issue when I try to run the evaluate_sbert.py in Colab. It was working all fine till yesterday. I have not made any change. Just pip installed beir, git cloned the beir repo and ran the python file without any change to the file. The error is something like this:

2021-11-10 15:30:08.003342: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2150, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/dist-packages/transformers/modeling_tf_utils.py", line 637, in <module>
    class TFPreTrainedModel(tf.keras.Model, TFModelUtilsMixin, TFGenerationMixin, PushToHubMixin):
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/lazy_loader.py", line 62, in __getattr__
    module = self._load()
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/lazy_loader.py", line 45, in _load
    module = importlib.import_module(self.__name__)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/dist-packages/keras/__init__.py", line 25, in <module>
    from keras import models
  File "/usr/local/lib/python3.7/dist-packages/keras/models.py", line 20, in <module>
    from keras import metrics as metrics_module
  File "/usr/local/lib/python3.7/dist-packages/keras/metrics.py", line 26, in <module>
    from keras import activations
  File "/usr/local/lib/python3.7/dist-packages/keras/activations.py", line 20, in <module>
    from keras.layers import advanced_activations
  File "/usr/local/lib/python3.7/dist-packages/keras/layers/__init__.py", line 23, in <module>
    from keras.engine.input_layer import Input
  File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_layer.py", line 21, in <module>
    from keras.engine import base_layer
  File "/usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py", line 43, in <module>
    from keras.mixed_precision import loss_scale_optimizer
  File "/usr/local/lib/python3.7/dist-packages/keras/mixed_precision/loss_scale_optimizer.py", line 18, in <module>
    from keras import optimizers
  File "/usr/local/lib/python3.7/dist-packages/keras/optimizers.py", line 26, in <module>
    from keras.optimizer_v2 import adadelta as adadelta_v2
  File "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/adadelta.py", line 22, in <module>
    from keras.optimizer_v2 import optimizer_v2
  File "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/optimizer_v2.py", line 37, in <module>
    "/tensorflow/api/keras/optimizers", "keras optimizer usage", "method")
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/monitoring.py", line 361, in __init__
    len(labels), name, description, *labels)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/monitoring.py", line 135, in __init__
    self._metric = self._metric_methods[self._label_length].create(*args)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Another metric with the same name already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2150, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/__init__.py", line 19, in <module>
    from . import (
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/layoutlm/__init__.py", line 22, in <module>
    from .configuration_layoutlm import LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP, LayoutLMConfig
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/layoutlm/configuration_layoutlm.py", line 22, in <module>
    from ...onnx import OnnxConfig, PatchingSpec
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__init__.py", line 17, in <module>
    from .convert import export, validate_model_outputs
  File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 23, in <module>
    from .. import PreTrainedModel, PreTrainedTokenizer, TensorType, TFPreTrainedModel, is_torch_available
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2140, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2154, in _get_module
    ) from e
RuntimeError: Failed to import transformers.modeling_tf_utils because of the following error (look up to see its traceback):
Another metric with the same name already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/content/beir/examples/retrieval/evaluation/dense/evaluate_sbert.py", line 2, in <module>
    from beir.retrieval import models
  File "/usr/local/lib/python3.7/dist-packages/beir/retrieval/models/__init__.py", line 1, in <module>
    from .sentence_bert import SentenceBERT
  File "/usr/local/lib/python3.7/dist-packages/beir/retrieval/models/sentence_bert.py", line 1, in <module>
    from sentence_transformers import SentenceTransformer
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/__init__.py", line 3, in <module>
    from .datasets import SentencesDataset, ParallelSentencesDataset
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/datasets/__init__.py", line 3, in <module>
    from .ParallelSentencesDataset import ParallelSentencesDataset
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/datasets/ParallelSentencesDataset.py", line 4, in <module>
    from .. import SentenceTransformer
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/SentenceTransformer.py", line 27, in <module>
    from .models import Transformer, Pooling, Dense
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/models/__init__.py", line 1, in <module>
    from .Transformer import Transformer
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/models/Transformer.py", line 2, in <module>
    from transformers import AutoModel, AutoTokenizer, AutoConfig
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2140, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 2154, in _get_module
    ) from e
RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback):
Failed to import transformers.modeling_tf_utils because of the following error (look up to see its traceback):
Another metric with the same name already exists.
nreimers commented 2 years ago

It might be connected to the recent tensorflow v2.7, it broke several things in huggingface transformers.

pritamdeka commented 2 years ago

@nreimers I guess it was due to the tensorflow update as the error seems fixed now so I am closing the issue.