search

yao8839836 / fast_text_gcn

FastGCN for inductive text classification
85 stars 22 forks source link

Memory issue on converting sparse to dense matrix #1

Open sankhoroy opened 5 years ago

sankhoroy commented 5 years ago

In https://github.com/yao8839836/fast_text_gcn/blob/master/pubmed_inductive_appr2layers.py line number 74 I have features sparse matrix of size 47,000 x 47,000.

On executing I am having memory error.

features = nontuple_preprocess_features(features).todense() train_features = features[tmp_index]

image

Error Mesage :

` features = nontuple_preprocess_features(features).todense() Traceback (most recent call last):

File "", line 1, in features = nontuple_preprocess_features(features).todense()

File "/home/hemanggoyal/anaconda/lib/python3.7/site-packages/scipy/sparse/base.py", line 848, in todense return asmatrix(self.toarray(order=order, out=out))

File "/home/hemanggoyal/anaconda/lib/python3.7/site-packages/scipy/sparse/compressed.py", line 1024, in toarray out = self._process_toarray_args(order, out)

File "/home/hemanggoyal/anaconda/lib/python3.7/site-packages/scipy/sparse/base.py", line 1186, in _process_toarray_args return np.zeros(self.shape, dtype=self.dtype, order=order)

MemoryError `

qycurious commented 4 years ago

Hello, I also have this problem. Have you solved this problem？

sankhoroy commented 4 years ago

We tried on smaller size matrices. It worked. But no permanent solution we have implemented.

caffe-in commented 2 years ago

I also have this problem. When I use 20ng, the code at line num 74 give a shape (61603, 61603) numpy matrix, which is so big that caused the memory error.