joachimwolff
commented
2 years ago Well, the error message is quite obvious numpy.core._exceptions.MemoryError: Unable to allocate 432. GiB for an array with shape (240707, 240707) and data type float64
Your system has too less memory, that's it. It tries to allocate 432 GB, but your system can't provide this. Use a machine with more memory.
Hi I am having an issue when runing the hicCorrectMatrix tool. If you could give me some advice/solutions, that would be great. I am running the tool on a computer cluster from my lab.
[ x] Paste your HiCExplorer version (
hicInfo --version) and your python version (python --version) below. hicInfo 3.6 Python 3.8.12[ x] Paste the full HiCExplorer command that produces the issue below hicCorrectMatrix correct --matrix hicsummatrix_AS-522121-LR-55516.h5 --correctionMethod ICE --chromosomes 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 X Y --perchr --verbose -o correct_matrix_AS-522121-LR-55516.h5 --filterThreshold -1.5 5
[ x] Paste the output printed on screen from the command that produces the issue below (ignore if you simply spotted the issue in the code/documentation).
INFO:hicexplorer.hicCorrectMatrix:Removing 99649 zero value bins INFO:hicexplorer.hicCorrectMatrix:matrix contains 268468270 data points. Sparsity 0.000. INFO:hicexplorer.iterativeCorrection:starting iterative correction INFO:hicexplorer.iterativeCorrection:pass 5 Estimated time 0:0:57 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.3051512105557296 INFO:hicexplorer.iterativeCorrection:pass 10 Estimated time 0:0:56 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.11059398752643257 INFO:hicexplorer.iterativeCorrection:pass 15 Estimated time 0:0:56 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.029765768175725738 INFO:hicexplorer.iterativeCorrection:pass 20 Estimated time 0:0:55 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.008627011652300842 INFO:hicexplorer.iterativeCorrection:pass 25 Estimated time 0:0:55 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.0024293316539122367 INFO:hicexplorer.iterativeCorrection:pass 30 Estimated time 0:0:54 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.0006854490202450325 INFO:hicexplorer.iterativeCorrection:pass 35 Estimated time 0:0:53 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 0.0001926682259609791 INFO:hicexplorer.iterativeCorrection:pass 40 Estimated time 0:0:53 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 5.4232375763074714e-05 INFO:hicexplorer.iterativeCorrection:pass 45 Estimated time 0:0:52 INFO:hicexplorer.iterativeCorrection:max delta - 1 = 1.6743503604921273e-05 INFO:hicexplorer.iterativeCorrection:[iterative correction] 49 iterations used
Traceback (most recent call last): File "/beegfs/home/lerat/.conda/envs/hicexplorer/bin/hicCorrectMatrix", line 7, in
main()
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/hicexplorer/hicCorrectMatrix.py", line 699, in main
corrected_matrix[chr_range[0]:chr_range[1],
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/scipy/sparse/lil.py", line 333, in setitem
IndexMixin.setitem(self, key, x)
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/scipy/sparse/_index.py", line 116, in setitem
self._set_arrayXarray_sparse(i, j, x)
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/scipy/sparse/lil.py", line 319, in _set_arrayXarray_sparse
x = np.asarray(x.toarray(), dtype=self.dtype)
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/scipy/sparse/coo.py", line 321, in toarray
B = self._process_toarray_args(order, out)
File "/beegfs/home/lerat/.conda/envs/hicexplorer/lib/python3.8/site-packages/scipy/sparse/base.py", line 1202, in _process_toarray_args
return np.zeros(self.shape, dtype=self.dtype, order=order)
numpy.core._exceptions.MemoryError: Unable to allocate 432. GiB for an array with shape (240707, 240707) and data type float64