search

4dn-dcic / hic2cool

Lightweight converter between hic and cool contact matrices.
MIT License
67 stars 7 forks source link

Multiprocessing Error #55

Open aakashsur opened 3 years ago

aakashsur commented 3 years ago

When I try to run hic2convert with more than one processor, I get the following error - 

Traceback (most recent call last):
  File "/Users/aakashsur/.virtual/default/bin/hic2cool", line 8, in <module>
    sys.exit(main())
  File "/Users/aakashsur/.virtual/default/lib/python3.8/site-packages/hic2cool/__main__.py", line 86, in main
    hic2cool_convert(args.infile, args.outfile, args.resolution, args.nproc, args.warnings, args.silent)
  File "/Users/aakashsur/.virtual/default/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 936, in hic2cool_convert
    tmp_chunk = parse_hic(req, pool, nproc, chr_key, unit, binsize,
  File "/Users/aakashsur/.virtual/default/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 359, in parse_hic
    result_all.extend(mpi_result[mpi].get())
  File "/Users/aakashsur/.python/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
NameError: name 'reqarr' is not defined

It runs normally when I just have one processor.

joachimwolff commented 2 years ago

I can confirm this, but getting a different error: 

hic2cool convert -r 5000 -p 10 matrix.hic matrix.cool
##########################
### hic2cool / convert ###
##########################
### Header info from hic
... Chromosomes:  ['ALL', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', 'M', 'X', 'Y']
... Resolutions:  [2500000, 1000000, 500000, 250000, 100000, 50000, 25000, 10000, 5000]
... Normalizations:  ['VC', 'VC_SQRT', 'KR']
... Genome:  mm10
### Converting
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 377, in build_counts_chunk
    records = read_block(req, block_record)
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 232, in read_block
    uncompressedBytes = zlib.decompress(compressedBytes)
zlib.error: Error -3 while decompressing data: invalid block type
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/bin/hic2cool", line 10, in <module>
    sys.exit(main())
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/site-packages/hic2cool/__main__.py", line 86, in main
    hic2cool_convert(args.infile, args.outfile, args.resolution, args.nproc, args.warnings, args.silent)
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 936, in hic2cool_convert
    tmp_chunk = parse_hic(req, pool, nproc, chr_key, unit, binsize,
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/site-packages/hic2cool/hic2cool_utils.py", line 359, in parse_hic
    result_all.extend(mpi_result[mpi].get())
  File "/wolfjoac/miniconda3/envs/hicexplorer_dev/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
zlib.error: Error -3 while decompressing data: invalid block type