search

vatlab / varianttools

software tool for the manipulation, annotation, selection, and analysis of variants in the context of next-gen sequencing analysis
https://vatlab.github.io/vat-docs/
GNU General Public License v3.0
31 stars 4 forks source link

Python 3.8 use spawn as default method to create subprocesses. #144

Open BoPeng opened 4 years ago

BoPeng commented 4 years ago

Perhaps I have some particular version of SWIG?

> vtools import test.vcf                                                            INFO: Using primary reference genome hg19 of the project.
INFO: Importing variants from test.vcf (1/1)
ERROR: cannot pickle 'SwigPyObject' object
Traceback (most recent call last):
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer.py", line 1024, in importVariants
    store.importGenotypes(importer)
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/geno_store.py", line 1855, in importGenotypes
    return importGenotypesInParallel(importer)
  File "/Users/bpeng/vatlab/vtools/src/variant_tools/importer_allele_hdf5.py", line 1517, in importGenotypesInParallel
BoPeng commented 4 years ago

Just tested python 3.6, and python 3.7 and they work just fine ... so the problem is only presented in python 3.8. Note that Python 3.8 uses a new protocol for Pickle as default.

BoPeng commented 4 years ago

The problem is caused by https://bugs.python.org/issue33725

Long story short: Newer versions of MacOSX does not like fork() and might cause problems. CPython guys understood this and change the default method to fork process with spawn.

Now, for fork we are NOT picking stuff like functions, and on spawn we are, then a lot of things will sudden fall apart.

BoPeng commented 4 years ago

This is not a clean fix since fork should be deprecated (at least as suggested by the core team), but it is a quick fix to make vtools work again with python 3.8.