search

allenai / bilm-tf

Tensorflow implementation of contextualized word representations from bi-directional language models
Apache License 2.0
1.62k stars 452 forks source link

Is there any version issue to train the bilm on new corpus #160

Closed anupamjamatia closed 5 years ago

anupamjamatia commented 5 years ago

I am not able to train the bilm for new corpus. I have gone through the https://github.com/allenai/bilm-tf but after giving the command python -m unittest discover tests/ to test the environment I am getting the following errors.

(keras-tf-env3) anupam@anupam-OMEN-HP:~/Desktop/ElMo/bilm-tf-master$ python -m unittest discover tests/ > rs.txt EEEE

ERROR: test_data (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_data Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import * File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name import(name) File "/home/anupam/Desktop/ElMo/bilm-tf-master/tests/test_data.py", line 7, in from bilm.data import UnicodeCharsVocabulary, Vocabulary, \ File "/home/anupam/Desktop/ElMo/bilm-tf-master/bilm/init.py", line 3, in from .model import BidirectionalLanguageModel, dump_token_embeddings, \ File "/home/anupam/Desktop/ElMo/bilm-tf-master/bilm/model.py", line 3, in import tensorflow as tf File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

====================================================================== ERROR: test_elmo (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_elmo Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import * File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name import(name) File "/home/anupam/Desktop/ElMo/bilm-tf-master/tests/test_elmo.py", line 7, in import tensorflow as tf File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

====================================================================== ERROR: test_model (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_model Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import * File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name import(name) File "/home/anupam/Desktop/ElMo/bilm-tf-master/tests/test_model.py", line 10, in import tensorflow as tf File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

====================================================================== ERROR: test_training (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_training Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import * File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name import(name) File "/home/anupam/Desktop/ElMo/bilm-tf-master/tests/test_training.py", line 8, in import tensorflow as tf File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/init.py", line 24, in from tensorflow.python import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in from tensorflow.python.pywrap_tensorflow_internal import File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/home/anupam/anaconda3/envs/keras-tf-env3/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcusolver.so.8.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions. Include the entire stack trace above this error message when asking for help.

Ran 4 tests in 0.000s

FAILED (errors=4)

Can you kindly help me please.

matt-peters commented 5 years ago

This is an issue with your CUDA installation and tensorflow. From your stack trace:

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions

anupamjamatia commented 5 years ago

Yes - I have seen the link and was trying to solve it out. But not able to configure the system. Do May I know what’s the requirement of version of Tensorflow, CUDA CUDcnn bazzel is required ?

On Tue, 29 Jan 2019 at 2:54 AM, Matthew Peters notifications@github.com wrote:

This is an issue with your CUDA installation and tensorflow. From your stack trace:

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/allenai/bilm-tf/issues/160#issuecomment-458307031, or mute the thread https://github.com/notifications/unsubscribe-auth/ALr0d8iJ0L2XoNX4E7Spsx62KTYMQrftks5vH2p7gaJpZM4aVvcb .

-- Regards Anupam Jamatia NIT, Agartala. CSED