Repeated MemoryError in each ForkPoolWorker

[amber4mint]

Hi,

Thank you for developing such a convenient program, CITE-seq-Count, which helped our lab a lot, so far.

But recently when I processed the fastq data (total 516,131,880 reads of R1 and R2), I continuedly get the message "MemoryError" in each ForkPoolWorker, which showed below:

Process ForkPoolWorker-7: Traceback (most recent call last): File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 131, in worker put((job, i, result)) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/queues.py", line 375, in put obj = _ForkingPickler.dumps(obj) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, *args, *kwds).dump(obj) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/dill/_dill.py", line 498, in dump StockPickler.dump(self, obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 487, in dump self.save(obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, rv) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 713, in save_reduce self._batch_setitems(dictitems) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 997, in _batch_setitems save(v) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 996, in _batch_setitems save(k) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 536, in save self.framer.commit_frame() File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 233, in commit_frame write(data) MemoryError

and the program seem not to be stopped, then continue the processing in next ForkPoolWork:

MemoryError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/process.py", line 315, in _bootstrap self.run() File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/process.py", line 108, in run self._target(*self._args, *self._kwargs) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 133, in worker wrapped = MaybeEncodingError(e, result[1]) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 86, in init self.value = repr(value) MemoryError Processed 1,000,000 reads in 7.0 hours, 54.0 minutes, 14.38 seconds. Total reads: 13,000,000 in child 718 Processed 1,000,000 reads in 15.0 hours, 51.0 minutes, 26.51 seconds. Total reads: 13,000,000 in child 727 Processed 1,000,000 reads in 1.0 day, 3.0 hours, 54.0 minutes, 43.42 seconds. Total reads: 11,000,000 in child 723 Processed 1,000,000 reads in 1.0 day, 4.0 hours, 24.0 minutes, 0.1068 seconds. Total reads: 11,000,000 in child 725 Processed 1,000,000 reads in 1.0 day, 6.0 hours, 19.0 minutes, 51.67 seconds. Total reads: 12,000,000 in child 720 Process ForkPoolWorker-24: Traceback (most recent call last): File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/pool.py", line 131, in worker put((job, i, result)) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/queues.py", line 375, in put obj = _ForkingPickler.dumps(obj) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, args, *kwds).dump(obj) File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/dill/_dill.py", line 498, in dump StockPickler.dump(self, obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 487, in dump self.save(obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 603, in save self.save_reduce(obj=obj, rv) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 692, in save_reduce save(args) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 886, in save_tuple save(element) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/site-packages/dill/_dill.py", line 990, in save_module_dict StockPickler.save_dict(pickler, obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 971, in save_dict self._batch_setitems(obj.items()) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 996, in _batch_setitems save(k) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 870, in save_str self.memoize(obj) File "/home/amber4mint/miniconda3/lib/python3.9/pickle.py", line 511, in memoize self.memo[id(obj)] = idx, obj MemoryError

The codes that I run: CITE-seq-Count -R1 /home/amber4mint/NGS1100905/KO_10X_210104_FB_S1_L002_R1_001.fastq.gz -R2 /home/amber4mint/NGS1100905/KO_10X_210104_FB_S1_L002_R2_001.fastq.gz -t /home/amber4mint/human_tag.csv -cbf 1 -cbl 16 -umif 17 -umil 26 -cells 10000 -o /home/amber4mint/cite-seq_210104/ --sliding-window -T 35

I set the threads to 35, because I met the 'struct.error' same as issue #75 and #93, and referred their solutions, before I got the 'MemoryError'.

Many thanks for your any advice!

Sincerely, Amber