search

roamanalytics / mittens

A fast implementation of GloVe, with optional retrofitting
Apache License 2.0
243 stars 31 forks source link

Cannot run mittens with tensorflow 2.1 #15

Closed richcmwang closed 4 years ago

richcmwang commented 4 years ago

After installing tensorflow 2.1, I cannt run GloVe any more - "fit" function gives the following errors:

Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor 2020-03-01 17:54:01.126392: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2020-03-01 17:54:01.127263: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.127355: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.127418: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128038: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128072: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128111: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128134: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128168: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.128184: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.129233: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-03-01 17:54:01.129268: W tensorflow/stream_executor/stream.cc:2041] attempting to perform BLAS operation using StreamExecutor without BLAS support Traceback (most recent call last): File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call return fn(*args) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn target_list, run_metadata) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Blas GEMM launch failed : a.shape=(4, 1), b.shape=(1, 4), m=4, n=4, k=1 [[{{node Tensordot_1/MatMul}}]] [[Sum/_5]] (1) Internal: Blas GEMM launch failed : a.shape=(4, 1), b.shape=(1, 4), m=4, n=4, k=1 [[{{node Tensordot_1/MatMul}}]] 0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/richard/Documents/AI/XCS224U/similarity_methods.py", line 28, in embeddings = glove_model.fit(cooccurrence) File "/home/richard/anaconda3/lib/python3.7/site-packages/mittens/mittens_base.py", line 240, in fit X, fixed_initialization=fixed_initialization) File "/home/richard/anaconda3/lib/python3.7/site-packages/mittens/mittens_base.py", line 84, in fit fixed_initialization=fixed_initialization) File "/home/richard/anaconda3/lib/python3.7/site-packages/mittens/tf_mittens.py", line 83, in _fit self.log_coincidence: log_coincidence}) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 960, in run run_metadata_ptr) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1183, in _run feed_dict_tensor, options, run_metadata) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1361, in _do_run run_metadata) File "/home/richard/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1386, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found. (0) Internal: Blas GEMM launch failed : a.shape=(4, 1), b.shape=(1, 4), m=4, n=4, k=1 [[node Tensordot_1/MatMul (defined at /anaconda3/lib/python3.7/site-packages/mittens/tf_mittens.py:151) ]] [[Sum/_5]] (1) Internal: Blas GEMM launch failed : a.shape=(4, 1), b.shape=(1, 4), m=4, n=4, k=1 [[node Tensordot_1/MatMul (defined at /anaconda3/lib/python3.7/site-packages/mittens/tf_mittens.py:151) ]] 0 successful operations. 0 derived errors ignored.

Original stack trace for 'Tensordot_1/MatMul': File "/Documents/AI/XCS224U/similarity_methods.py", line 28, in embeddings = glove_model.fit(cooccurrence) File "/anaconda3/lib/python3.7/site-packages/mittens/mittens_base.py", line 240, in fit X, fixed_initialization=fixed_initialization) File "/anaconda3/lib/python3.7/site-packages/mittens/mittens_base.py", line 84, in fit fixed_initialization=fixed_initialization) File "/anaconda3/lib/python3.7/site-packages/mittens/tf_mittens.py", line 56, in _fit self._build_graph(vocab, initial_embedding_dict) File "/anaconda3/lib/python3.7/site-packages/mittens/tf_mittens.py", line 151, in _build_graph tf.tensordot(self.bw, tf.transpose(self.ones), axes=1) + \ File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 4106, in tensordot ab_matmul = matmul(a_reshape, b_reshape) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper return target(*args, **kwargs) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 2798, in matmul a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 5626, in mat_mul name=name) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 742, in _apply_op_helper attrs=attr_protos, op_def=op_def) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3322, in _create_op_internal op_def=op_def) File "/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1756, in init self._traceback = tf_stack.extract_stack()

richcmwang commented 4 years ago

double submissions - please see previous issue.