medaka smolecule concurrent.futures.process.BrokenProcessPool

[rob234king]

Describe the bug When I run medaka smolecule I often get this error, a few data sets work but usually its this error. 2 of the 20 samples I have go through fine but the rest all fail like this. I think it's a bug with medaka rather than my command or data. I have python 3.8 in the conda environment.

Logging

[07:35:05 - DataIndx] Loaded 1/1 (100.00%) sample files.
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/process.py", line 394, in _queue_management_worker
    work_item.future.set_exception(bpe)
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 547, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2ac3f09ac910 state=cancelled>
Traceback (most recent call last):
  File "/home/rking/miniconda3/envs/UMI/bin/medaka", line 11, in <module>
    sys.exit(main())
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/medaka.py", line 720, in main
    args.func(args)
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/smolecule.py", line 428, in main
    medaka.stitch.stitch(args)
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/stitch.py", line 231, in stitch
    for (rname, start, stop), seq_parts, qualities in contigs:
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/site-packages/medaka/stitch.py", line 157, in collapse_neighbours
    for contig in contigs:
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/home/rking/miniconda3/envs/UMI/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument

Environment (if you do not have a GPU, write No GPU):

• not using GPU, medaka 1.6.1, conda install

Additional context using UMI amplicons with medaka smolecule.

[rob234king]

I don't get the error if I split the files up to 100 pieces then run medaka on each. So got a workaround at least.

[rob234king]

actually no this is still failing for some files with that future error even chopping up the input. For one set of data it went through fine. now I'm trying another and it's failing at the medaka stage even though dividing up to 100 files and 1 cpu per job. I've set TMPDIR to local location now and increased mem per job to 2G, so see if that has an effect. Normally tool issues are tmp or mem but it might be more annoying.

[gordonrix]

Were you able to find a reliable workaround for this? I'm struggling to get around this issue

[cjw85]

@rob234king

Please try v1.7.0, which contains some amendments to the smolecule command to fix this issue in at least some circumstnaces.