MemoryError - Githubissues

kfuku52 commented 3 years ago

Those errors occur in a large dataset, typically a tree with >300 leaves.

Traceback (most recent call last):
  File "/opt/conda/envs/python3/bin/csubst", line 283, in <module>
    args.handler(args)
  File "/opt/conda/envs/python3/bin/csubst", line 47, in command_analyze
    main_analyze(g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 273, in main_analyze
    g = cb_search(g, b, S_tensor, N_tensor, id_combinations, mode='foreground', write_cb=True)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 108, in cb_search
    cb, g = omega.calc_omega(cb, S_tensor, N_tensor, g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 294, in calc_omega
    cb = get_E(cb, g, N_tensor, S_tensor)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 193, in get_E
    EN_tensor = substitution.get_substitution_tensor(state_pepE, g['state_pep'], mode='asis', g=g, mmap_attr='EN')
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/substitution.py", line 60, in get_substitution_tensor
    sub_tensor = numpy.nan_to_num(sub_tensor, nan=0)
  File "<__array_function__ internals>", line 5, in nan_to_num
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/numpy/lib/type_check.py", line 480, in nan_to_num
    idx_neginf = isneginf(d)
  File "<__array_function__ internals>", line 5, in isneginf
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/numpy/lib/ufunclike.py", line 53, in func
    return f(x, out=out, **kwargs)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/numpy/lib/ufunclike.py", line 266, in isneginf
    return nx.logical_and(is_inf, signbit, out)
numpy.core._exceptions.MemoryError: Unable to allocate 149. MiB for an array with shape (981, 399, 1, 20, 20) and data type bool

kfuku52 commented 3 years ago

Traceback (most recent call last):
  File "/opt/conda/envs/python3/bin/csubst", line 283, in <module>
    args.handler(args)
  File "/opt/conda/envs/python3/bin/csubst", line 47, in command_analyze
    main_analyze(g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 273, in main_analyze
    g = cb_search(g, b, S_tensor, N_tensor, id_combinations, mode='foreground', write_cb=True)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 108, in cb_search
    cb, g = omega.calc_omega(cb, S_tensor, N_tensor, g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 294, in calc_omega
    cb = get_E(cb, g, N_tensor, S_tensor)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 193, in get_E
    EN_tensor = substitution.get_substitution_tensor(state_pepE, g['state_pep'], mode='asis', g=g, mmap_attr='EN')
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/substitution.py", line 60, in get_substitution_tensor
    sub_tensor = numpy.nan_to_num(sub_tensor, nan=0)
  File "<__array_function__ internals>", line 5, in nan_to_num
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/numpy/lib/type_check.py", line 461, in nan_to_num
    x = _nx.array(x, subok=True, copy=copy)
numpy.core._exceptions.MemoryError: Unable to allocate 1.46 GiB for an array with shape (1317, 373, 1, 20, 20) and data type float64

kfuku52 commented 3 years ago

Traceback (most recent call last):
  File "/opt/conda/envs/python3/bin/csubst", line 283, in <module>
    args.handler(args)
  File "/opt/conda/envs/python3/bin/csubst", line 47, in command_analyze
    main_analyze(g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 273, in main_analyze
    g = cb_search(g, b, S_tensor, N_tensor, id_combinations, mode='foreground', write_cb=True)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/main_analyze.py", line 108, in cb_search
    cb, g = omega.calc_omega(cb, S_tensor, N_tensor, g)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 294, in calc_omega
    cb = get_E(cb, g, N_tensor, S_tensor)
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/omega.py", line 193, in get_E
    EN_tensor = substitution.get_substitution_tensor(state_pepE, g['state_pep'], mode='asis', g=g, mmap_attr='EN')
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/csubst/substitution.py", line 60, in get_substitution_tensor
    sub_tensor = numpy.nan_to_num(sub_tensor, nan=0)
  File "<__array_function__ internals>", line 5, in nan_to_num
  File "/opt/conda/envs/python3/lib/python3.9/site-packages/numpy/lib/type_check.py", line 478, in nan_to_num
    idx_nan = isnan(d)
numpy.core._exceptions.MemoryError: Unable to allocate 183. MiB for an array with shape (271, 1774, 1, 20, 20) and data type bool

kfuku52 commented 3 years ago

In my case with SGE, increasing allocated virtual memory like -l s_vmem 64G solved the problem.

kfuku52 / csubst

MemoryError #18