OverflowError: cannot serialize a bytes object larger than 4 GiB

[ShuaiC-CYLab]

When I use multiprocessing on a big data, I got the follow error. How could I solve this problem?

/home/black/anaconda3/lib/python3.7/site-packages/dask/dataframe/utils.py:13: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm
Loaded expression matrix of 71366 cells and 25216 genes in 23.77125358581543 seconds...
Loaded 1839 TFs...
starting genie3 using 10 processes...
  0%|                                                 | 0/25216 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/black/anaconda3/bin/arboreto_with_multiprocessing.py", line 198, in <module>
    main()
  File "/home/black/anaconda3/bin/arboreto_with_multiprocessing.py", line 184, in main
    total=len(gene_names),
  File "/home/black/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 937, in __iter__
    for obj in iterable:
  File "/home/black/anaconda3/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
  File "/home/black/anaconda3/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/black/anaconda3/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/black/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
**OverflowError: cannot serialize a bytes object larger than 4 GiB**
Exception ignored in: <function tqdm.__del__ at 0x7f1da49cd840>
Traceback (most recent call last):
  File "/home/black/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 889, in __del__
  File "/home/black/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1095, in close
  File "/home/black/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py", line 441, in _decr_instances
  File "/home/black/anaconda3/lib/python3.7/_weakrefset.py", line 109, in remove
KeyError: <weakref at 0x7f1da0953228; to 'tqdm' at 0x7f1da09840b8>

[ghuls]

Could you try with the lastest pySCENIC docker/podman/singularity/apptainer image: https://pyscenic.readthedocs.io/en/latest/installation.html#docker-podman-and-singularity-apptainer-images

[ShuaiC-CYLab]

Could you try with the lastest pySCENIC docker/podman/singularity/apptainer image: https://pyscenic.readthedocs.io/en/latest/installation.html#docker-podman-and-singularity-apptainer-images

Thanks for your reply, I have solved this problem by running pySCENIC on a HPC. And I will close this comment.