OpenGPTX / lm-evaluation-harness

A framework for few-shot evaluation of autoregressive language models.
MIT License
8 stars 8 forks source link

`MegatronLMClient` expects cache usage #106

Open janEbert opened 7 months ago

janEbert commented 7 months ago

When trying to issue a length-exceeded warning while --no_cache is given, the MegatronLMClient errors out with the below stack trace. The reason being that cache_key is None, which is not checked. The easiest solution would probably be to just omit printing the text strings if cache_key is None (i.e., if caching is disabled).

Traceback (most recent call last):
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/main.py", line 101, in <module>
    main()
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/main.py", line 67, in main
    results = evaluator.simple_evaluate(
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
    return fn(*args, **kwargs)
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/evaluator.py", line 103, in simple_evaluate
    results = evaluate(
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
    return fn(*args, **kwargs)
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/evaluator.py", line 297, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/base.py", line 210, in loglikelihood_rolling
    string_nll = self._loglikelihood_tokens(
  File "/p/project/opengptx-elm/ebert1/opengpt/lm-evaluation-harness/lm_eval/models/megatronlm.py", line 187, in _loglikelihood_tokens
    f"WARNING: Length of concatenated context ...{repr(cache_key[0][-20:])} and continuation {repr(cache_key[1])} exceeds max length {self.max_length + 1}"
TypeError: 'NoneType' object is not subscriptable