search

AlexTISYoung / snipar

Imputation of parental genotypes, inference of sibling IBD segments, family based GWAS, and polygenic score analyses.
MIT License
23 stars 4 forks source link

AttributeError: 'NoneType' object has no attribute 'dumps' when running impute_runner.py #2

Closed yystat closed 4 years ago

yystat commented 4 years ago

Dear Prof.,

Sorry to bother you again. When I'm running the following example codes (from https://sibreg.readthedocs.io/en/master/tutorial.html)

python impute_runner.py 1 2 test_data/sample.segments.gz test_data/sample1 --king test_data/sample.king --agesex test_data/sample.agesex --out_prefix test_data/sample --threads 4

It could successfully impute chromosome 1. However, after written the result for chromosome 1, an error occurred and the process stopped. The error message is:

Traceback (most recent call last):
  File "/s/pkg/linux64/python/3.8.3/lib/python3.8/multiprocessing/pool.py", line 268, in __del__
  File "/s/pkg/linux64/python/3.8.3/lib/python3.8/multiprocessing/queues.py", line 362, in put
AttributeError: 'NoneType' object has no attribute 'dumps'

I barely know python and I don't know what causes this problem. Does it relate to the codes on line 195 for chromosome in range(args.from_chr, args.to_chr) and line 199 consumed_time = pool.map(run_imputation, inputs) in impute_runner.py? Could you please help me with this?

Of course, the above codes are supposed to impute chromosome 1 only and an error message after the job's done may not really matter. I also tried change 1 2 to 1 3 and it could impute chrs 1 and 2, and after than same error message.

Thank you very much!

Josh

AlexTISYoung commented 4 years ago

Hi Josh,

Thanks for your email. I am copying in Moeen, who wrote the imputation code, to see if he can help you.

Alex.

On Thu, 16 Jul 2020 at 21:15, Josh notifications@github.com wrote:

Dear Prof.,

Sorry to bother you again. When I'm running the following example codes (from https://sibreg.readthedocs.io/en/master/tutorial.html)

python impute_runner.py 1 2 test_data/sample.segments.gz test_data/sample1 --king test_data/sample.king --agesex test_data/sample.agesex --out_prefix test_data/sample --threads 4

It could successfully impute chromosome 1. However, after written the result for chromosome 1, an error occurred and the process stopped. The error message is:

Traceback (most recent call last): File "/s/pkg/linux64/python/3.8.3/lib/python3.8/multiprocessing/pool.py", line 268, in del File "/s/pkg/linux64/python/3.8.3/lib/python3.8/multiprocessing/queues.py", line 362, in put AttributeError: 'NoneType' object has no attribute 'dumps'

I barely know python and I don't know what causes this problem. Does it relate to the codes on line 195 for chromosome in range(args.from_chr, args.to_chr) and line 199 consumed_time = pool.map(run_imputation, inputs) in impute_runner.py? Could you please help me with this?

Of course, the above codes are supposed to impute chromosome 1 only and an error message after the job's done may not really matter. I also tried change 1 2 to 1 3 and it could impute chrs 1 and 2, and after than same error message.

Thank you very much!

Josh

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/AlexTISYoung/SNIPar/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQQS6OMUH2DWVTIFYK2XZTR37F6HANCNFSM4O5PG77A .

MoeenNehzati commented 4 years ago

Hi Josh, I looked the error up and it's probably due to some inconsistencies between installed packages. Have you installed it on a fresh virtual environment? This way you won't experience any problem related to other installed packages. Can you give installing on a fresh virtual environment a try(if you haven't already)? Aside from that, could you provide me with the output of "pip freeze" command on your computer? Thanks! Moeen

yystat commented 4 years ago

Thank you Moeen!

I'm using my department's server and I don't think I could install it on a virtual environment. The IT person installed Python3.8 for me. Below are the outputs of the pip freeze command:

Babel==0.9.6
backports-abc==0.5
backports.ssl-match-hostname==3.5.0.1
cffi==1.6.0
chardet==2.2.1
configobj==4.7.2
configshell-fb==1.1.25
cryptography==1.7.2
cupshelpers==1.0
decorator==3.4.0
dnspython==1.12.0
docopt==0.6.2
ebs==0.0.10
enum34==1.0.4
ethtool==0.8
fontforge==1.0
futures==3.1.1
gssapi==1.2.0
idna==2.4
iniparse==0.4
ipaclient==4.6.6
ipaddress==1.0.16
ipalib==4.6.6
ipaplatform==4.6.6
ipapython==4.6.6
IPy==0.75
javapackages==1.0.0
Jinja2==2.7.2
jwcrypto==0.4.2
kitchen==1.1.1
kmod==0.1
langtable==0.0.31
lxml==3.2.1
MarkupSafe==0.11
msgpack==0.6.2
natsort==5.4.1
netaddr==0.7.5
netifaces==0.10.4
perf==0.1
plinkio==0.9.7
ply==3.4
policycoreutils-default-encoding==0.1
psutil==5.6.7
py-rrdtool==0.2.2
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycparser==2.14
pycrypto==2.6.1
pycups==1.9.63
pycurl==7.19.0
pyfaidx==0.5.5
pygobject==3.22.0
pygpgme==0.3
pyliblzma==0.5.3
pyparsing==1.5.6
pysmbc==1.0.13
python-augeas==0.5.0
python-dateutil==1.5
python-ldap==2.4.15
python-linux-procfs==0.4.9
python-nss==0.16.0
python-yubico==1.2.3
pyudev==0.15
pyusb==1.0.0b1
pyxattr==0.5.1
PyYAML==3.11
pyzmq==15.3.0
qrcode==5.0.1
requests==2.6.0
rtslib-fb==2.1.69
salt==3000.3
schedutils==0.4
seobject==0.1
sepolicy==1.1
singledispatch==3.4.0.3
six==1.9.0
slip==0.4.0
slip.dbus==0.4.0
snpflip==0.0.6
SSSDConfig==1.16.4
targetcli-fb===2.1.fb49
tornado==4.2.1
urlgrabber==3.10
urllib3==1.10.2
urwid==1.1.1
yum-langpacks==0.4.2
yum-metadata-parser==1.1.4
MoeenNehzati commented 4 years ago

We have tested the package on python 3.6 so that might also be an issue. Am I safe to assume you can install new packages?

yystat commented 4 years ago

Yes, I think so.

MoeenNehzati commented 4 years ago

Ok. It means you should be able to create virtual environments. You just need to install venv first(via pip install venv). Create a clean virtual environment, preferably a python3.6 environment, using venv, and then use that for working with the package. python3.6 -m venv name-of-venv-directory should give you a python3.6 environment. After this whenever you enter source name-of-venv-directory/bin/activate the default python your terminal uses will become your virtual environment.

yystat commented 4 years ago

Thanks. It seems that I could not install it. If 3.6 works, then I could ask the IT person to install 3.6 or even 3.7 for me.

yystat commented 4 years ago

Our server has python3.6.5, and the installation has errors when I run python setup.py pytest (in total there are 8 failed, 6 passed, 3 warnings):

subprocess.CalledProcessError: Command '['python3.6', 'impute_runner.py', '1', '3', 'test_data/sample.segments.gz', 'test_data/sample~', '--king', 'test_data/sample.king', '--agesex', 'test_data/sample.agesex', '--out_prefix', 'outputs/tmp/test_sample_imputed']' returned non-zero exit status 1.

ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /misc/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/pysnptools/snpreader/wrap_plink_parser.cpython-36m-x86_64-linux-gnu.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "impute_runner.py", line 199, in <module>
    consumed_time = pool.map(run_imputation, inputs)
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /misc/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/pysnptools/snpreader/wrap_plink_parser.cpython-36m-x86_64-linux-gnu.so)

Then I ran bash tests/generate_test_population.sh and got the following error message which should be due to the same reason:

Traceback (most recent call last):
  File "example/simulate_trait_quad.py", line 33, in <module>
    gts = gts_f.read().val
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/pysnptools/snpreader/snpreader.py", line 460, in read
    val = self._read(None, None, order, dtype, force_python_only, view_ok)
  File "/s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/pysnptools/snpreader/bed.py", line 244, in _read
    from pysnptools.snpreader import wrap_plink_parser
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /s/pkg/linux64/python/3.6.5/lib/python3.6/site-packages/pysnptools/snpreader/wrap_plink_parser.cpython-36m-x86_64-linux-gnu.so)
MoeenNehzati commented 4 years ago

Apparently you are having problems with pysnptools(one of the dependencies). I'm not sure but I might be due to having an old version of libgcc. Can you show me the output of this command? strings /lib64/libstdc++.so.6 | grep GLIBCXX

yystat commented 4 years ago

yes, below are the results:

GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_DEBUG_MESSAGE_LENGTH
MoeenNehzati commented 4 years ago

It's aligned with what I suspected. The dependencies need GLIBCXX_3.4.21 but you only have up to GLIBCXX_3.4.19. This should be helpful. I know that you don't have admin access so maybe the solution with conda would solve your problem.