search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.9k stars 1.84k forks source link

Followed the documentation but its not working. Details below - #748

Closed abhibarman closed 1 year ago

abhibarman commented 1 year ago

When running below snippet -

python main.py \ --model hf-causal \ --model_args pretrained=EleutherAI/gpt-j-6B \ --tasks hellaswag \ --device cuda:0

Getting this error -

/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py:513: FutureWarning: In the future np.object will be defined as the corresponding NumPy scalar. np.object, Traceback (most recent call last): File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/init.py", line 42, in tf from tensorboard.compat import notf # noqa: F401 ImportError: cannot import name 'notf' from 'tensorboard.compat' (/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/init.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/main.py", line 6, in from lm_eval import tasks, evaluator, utils File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/lm_eval/tasks/init.py", line 5, in import lm_eval.base File "/home/ubuntu/abhijit/LLM_Eval/lm-evaluation-harness/lm_eval/base.py", line 14, in from accelerate import find_executable_batch_size File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/init.py", line 3, in from .accelerator import Accelerator File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/accelerator.py", line 39, in from .tracking import LOGGER_TYPE_TO_CLASS, GeneralTracker, filter_trackers File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/accelerate/tracking.py", line 42, in from torch.utils import tensorboard File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/init.py", line 12, in from .writer import FileWriter, SummaryWriter # noqa: F401 File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/writer.py", line 16, in from ._embedding import ( File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/utils/tensorboard/_embedding.py", line 9, in _HAS_GFILE_JOIN = hasattr(tf.io.gfile, "join") File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 65, in getattr return getattr(load_once(self), attr_name) File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 97, in wrapper cache[arg] = f(arg) File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/lazy.py", line 50, in load_once module = load_fn() File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorboard/compat/init.py", line 45, in tf import tensorflow File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/init.py", line 41, in from tensorflow.python.tools import module_util as _module_util File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/init.py", line 46, in from tensorflow.python import data File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/init.py", line 25, in from tensorflow.python.data import experimental File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/init.py", line 96, in from tensorflow.python.data.experimental import service File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/service/init.py", line 140, in from tensorflow.python.data.experimental.ops.data_service_ops import distribute File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/data_service_ops.py", line 25, in from tensorflow.python.data.experimental.ops import compression_ops File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/compression_ops.py", line 20, in from tensorflow.python.data.util import structure File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/util/structure.py", line 26, in from tensorflow.python.data.util import nest File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/data/util/nest.py", line 41, in from tensorflow.python.framework import sparse_tensor as _sparse_tensor File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/sparse_tensor.py", line 29, in from tensorflow.python.framework import constant_op File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py", line 29, in from tensorflow.python.eager import execute File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 27, in from tensorflow.python.framework import dtypes File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py", line 513, in np.object, File "/home/ubuntu/miniconda3/lib/python3.9/site-packages/numpy/init.py", line 319, in getattr raise AttributeError(__former_attrs__[attr]) AttributeError: module 'numpy' has no attribute 'object'. np.object was a deprecated alias for the builtin object. To avoid this error in existing code, use object by itself. Doing this will not modify any behavior and is safe. The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

haileyschoelkopf commented 1 year ago

Hi, I've not seen this error before but it looks like a problem with tensorboard, which is not a dependency on our end.

This should be fixed by doing pip install numpy==1.23 I believe.

Could you run pip freeze and paste that output here? That should provide some insight.

haileyschoelkopf commented 1 year ago

Closing as I can't replicate this and changing numpy version should fix this, please reopen if this issue persists!