search

griffithlab / pVACtools

http://www.pvactools.org
BSD 3-Clause Clear License
137 stars 59 forks source link

Can't run pvacseq: Illegal instruction (core dumped) #351

Closed serge2016 closed 5 years ago

serge2016 commented 5 years ago

According to https://pvactools.readthedocs.io/en/latest/pvacseq/run.html, there should exist some kind of help for pvacseq: pvacseq run -h. But it craches...

bio@bio-server:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:    18.04
Codename:   bionic
bio@bio-server:~$ docker -v
Docker version 18.09.3, build 774a1f4
bio@bio-server:~$ docker pull griffithlab/pvactools:1.3.5
1.3.5: Pulling from griffithlab/pvactools
Digest: sha256:4a920908315caec770af44298143f2a121a1a988bbcd390ecf44ad5c236284be
Status: Image is up to date for griffithlab/pvactools:1.3.5
bio@bio-server:~$ docker run --rm -ti --entrypoint /bin/bash griffithlab/pvactools:1.3.5
(base) root@ad1e3889215a:/opt/iedb# pvacseq run -h
Illegal instruction (core dumped)
(base) root@ad1e3889215a:/opt/iedb# pvacseq --help
Illegal instruction (core dumped)
(base) root@ad1e3889215a:/opt/iedb# pvactools -v
1.3.5
(base) root@ad1e3889215a:/opt/iedb# pvactools --help
usage: pvactools [-h] [-v] {download_cwls} ...

positional arguments:
  {download_cwls}
    download_cwls  Run pVACtools CWLs

optional arguments:
  -h, --help       show this help message and exit
  -v, --version    Display the currently installed pvactools version (default:
                   False)
(base) root@ad1e3889215a:/opt/iedb#
susannasiebert commented 5 years ago

A similar problem was reported in https://github.com/griffithlab/pVACtools/issues/331. As far as we can tell, this is system specific, i.e. we are unable to reproduce this on macOS or our Linux cluster. You might want to try and rebuild pvactools from scratch on your system and then make a docker container from it: 1) Build a .whl distribution file (https://packaging.python.org/tutorials/packaging-projects/#generating-distribution-archives)

git clone https://github.com/griffithlab/pVACtools.git
cd pVACtools
python setup.py bdist_wheel

This will create a .whl file in the distsubdirectory 2) Make your own docker container 2a) Clone the repo

cd ..
git clone https://github.com/griffithlab/docker-pvactools.git
cd docker-pvactools
cp ../pVACtools/dist/pvactools-1.3.5-py3-none-any.whl .

2b) Edit the Dockerfile

Stikus commented 5 years ago

Hello! I have the same error on CentOS!

[root@slurm2 ~]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
[root@slurm2 ~]# docker -v
Docker version 18.09.1, build 4c52b90
[root@slurm2 ~]# docker pull griffithlab/pvactools:1.3.5
1.3.5: Pulling from griffithlab/pvactools
Digest: sha256:4a920908315caec770af44298143f2a121a1a988bbcd390ecf44ad5c236284be
Status: Image is up to date for griffithlab/pvactools:1.3.5
[root@slurm2 ~]# docker run --rm -ti --entrypoint /bin/bash griffithlab/pvactools:1.3.5
(base) root@0f639a3e826d:/opt/iedb# pvacseq run -h
Illegal instruction (core dumped)
(base) root@0f639a3e826d:/opt/iedb# pvacseq --help
Illegal instruction (core dumped)
(base) root@0f639a3e826d:/opt/iedb# pvactools -v
1.3.5
(base) root@0f639a3e826d:/opt/iedb# pvactools --help
usage: pvactools [-h] [-v] {download_cwls} ...

positional arguments:
  {download_cwls}
    download_cwls  Run pVACtools CWLs

optional arguments:
  -h, --help       show this help message and exit
  -v, --version    Display the currently installed pvactools version (default:
                   False)
(base) root@0f639a3e826d:/opt/iedb#
serge2016 commented 5 years ago

The problem is with tensorflow:

(base) root@f32c56624a3a:/opt/iedb# python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Illegal instruction (core dumped)

If I downgrade the tensorflow to 1.5:

pip install --upgrade tensorflow==1.5

then everything works fine!

https://stackoverflow.com/questions/53950186/tensorflow-library-was-compiled-to-use-sse4-1-instructions-but-these-arent-ava

The question is why this problem exists inside Docker Image/Container?!!!

@susannasiebert, is it critical to have tensorflow 1.13.1? Maybe it is better to downgrade it to 1.5 for better compatibility with older CPUs?

malachig commented 5 years ago

I was unable to reproduce this problem with the docker image (1.3.5). It worked on our local cluster, on my mac laptop, and on AWS EC2 (latest Ubuntu) version.

This is just speculation, but it seems like maybe a hardware problem. It seems that more recent versions of Tensorflow are incompatible with certain CPUs (must be recompiled on that hardware).

For examples, see this discussion here: https://github.com/home-assistant/home-assistant/issues/18435 (CPUs lacking SSE4.1 instructions ) https://github.com/tensorflow/tensorflow/issues/17411 (CPUs lacking AVX instructions) https://tech.amikelive.com/node-887/how-to-resolve-error-illegal-instruction-core-dumped-when-running-import-tensorflow-in-a-python-program/ https://stackoverflow.com/questions/49657983/why-cant-tensorflow-be-imported

jackyenbioinfo commented 5 years ago

@serge2016 Thanks! downgraded to tensorflow 1.5 and it fixed my issue. I had no idea that it was the tensorflow that was causing the trouble! Thanks for sharing

susannasiebert commented 5 years ago

The latest version (1.3.7) pins tensorflow to 1.5.0 during install which should solve this problem. Please note that this might lead to installation problems on new python versions which can be resolved by installing tensorflow manually from .whl files available on google storage (https://www.tensorflow.org/install/pip). Closing this issue.