search

simonhmartin / genomics_general

General tools for genomic analyses.
341 stars 93 forks source link

freq.py error #65

Open Wennie-s opened 2 years ago

Wennie-s commented 2 years ago

Hi, when I sun freq.py, there is an error： Process Process-1: Traceback (most recent call last): File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/data/user003/soft/dfs/genomics_general-0.4/freq.py", line 55, in freqs_wrapper baseFreqs = popAlns[pop].siteFreqs(asCounts=asCounts) File "/data/user003/soft/dfs/genomics_general-0.4/genomics.py", line 1008, in siteFreqs return np.array([binBaseFreqs(self.numArray[:,x][self.nanMask[:,x]], asCounts=asCounts) for x in sites]) File "/data/user003/soft/dfs/genomics_general-0.4/genomics.py", line 576, in binBaseFreqs if asCounts: return np.bincount(numArr, minlength=4) MemoryError 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written 37904 slices queued | 37903 slices analysed | 37903 slices written | 106800441 lines written

When I reduced threads to 1, the same error still appears. My genome is very large (10 Gb) and I don't know why this happen. Can you help me?

simonhmartin commented 2 years ago

Hi, I'm sorry I missed this issue back in 2021. Did you ever manage to solve it? I don't think this was a true memory error but rather an issue with unrecognised genotypes in the input file.