search

EleutherAI / gpt-neo

An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.
https://www.eleuther.ai
MIT License
8.2k stars 945 forks source link

`--check-dataset` fails `tf.enable_eager_execution must be called at program startup` #159

Closed afiaka87 closed 3 years ago

afiaka87 commented 3 years ago 
python3 main.py --model gpt3_XL_256_Pile --check_dataset
2021-03-23 01:49:48.909831: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-03-23 01:49:48.909862: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From /opt/conda/envs/gptx/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Current step 400000
Traceback (most recent call last):
  File "main.py", line 257, in <module>
    main(args)
  File "main.py", line 85, in main
    check_dataset(input_fn, params, global_step=current_step)
  File "/root/Repo/GPTNeo/utils.py", line 206, in check_dataset
    tf.enable_eager_execution()
  File "/opt/conda/envs/gptx/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 5863, in enable_eager_execution
    return enable_eager_execution_internal(
  File "/opt/conda/envs/gptx/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 5925, in enable_eager_execution_internal
    raise ValueError(
ValueError: tf.enable_eager_execution must be called at program startup.

All other details are the same as in my issue here: https://github.com/EleutherAI/gpt-neo/issues/158

afiaka87 commented 3 years ago

I've got nothing to add to this one now for the same reason I listed at the bottom of issue 158. Feel free to re-open if you think it will help others.