search

ccdmb / predector

Effector prediction pipeline based on protein properties.
Apache License 2.0
11 stars 7 forks source link

Deepsig fails with missing tensorflow module #52

Closed fmobegi closed 2 years ago

fmobegi commented 3 years ago

Error message

Error executing process > 'deepsig (2)'

Caused by:
  Process `deepsig (2)` terminated with an error exit status (1)

Command executed:

  CHUNKSIZE="$(decide_task_chunksize.sh in.fasta "4" 100)"

  run () {
      set -e
      OUT="tmp$$"
      deepsig.py -f $1 -k euk -o "${OUT}" 1>&2
      cat "${OUT}"
      rm -f "${OUT}"
  }

  export -f run

  parallel         --halt now,fail=1         --joblog log.txt         -j "4"         -N "${CHUNKSIZE}"         --line-buffer          --recstart '>'         --cat run     < in.fasta     | cat > out.txt

  predutils r2js         --pipeline-version "1.1.0-beta"         -o out.ldjson         deepsig out.txt

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  Traceback (most recent call last):
    File "/opt/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/opt/conda/envs/predector/share/deepsig-0f1e1d9-0/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
    File "/opt/conda/envs/predector/lib/python3.7/site-packages/keras/__init__.py", line 21, in <module>
      from tensorflow.python import tf2
  ModuleNotFoundError: No module named 'tensorflow'
  Traceback (most recent call last):
    File "/opt/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/opt/conda/envs/predector/share/deepsig-0f1e1d9-0/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
    File "/opt/conda/envs/predector/lib/python3.7/site-packages/keras/__init__.py", line 21, in <module>
      from tensorflow.python import tf2
  ModuleNotFoundError: No module named 'tensorflow'
  parallel: This job failed:
  run $PARALLEL_TMP
darcyabjones commented 3 years ago

Thanks Frederick,

So the issue here is caused by the keras package no-longer depending on tensorflow. I tried the new deepsig release which has a bioconda package but they've changed the output format to only print positive predictions, so it only reports the score of proteins with predicted signal peptides :(

I'm just going to pin tensorflow as a dependency, and when we update the model we'll also remove deepsig altogether because it's not actually contributing anything.

darcyabjones commented 3 years ago

Hey Frederick!

This should now be fixed with 1.1.0-beta.1 Sorry for the trouble.

I've tried to pin a few dependencies so that this doesn't happen too often. It was also a good chance to deal with some other bugs I've known about for a while :)

Let me know when/if you have a chance to re-run the pipeline and if it works for you.

fmobegi commented 3 years ago

Thanks @darcyabjones. The pipeline now runs well.

darcyabjones commented 3 years ago

Thanks Fredrick!

TCHeaven commented 2 years ago

I recieve this error with 1.2.6

darcyabjones commented 2 years ago

Hi @TCHeaven,

I've just tried installing and using the pipeline and using it, and didn't encounter any issues. Can you please share some more details of your environment (e.g. is it conda? what operating system?) and a copy of the specific error you get?

If you are using conda, could you please activate the predector environment and share the output of conda list with me?

Thanks, Darcy

TCHeaven commented 2 years ago

Hey @darcyabjones The environment is conda, operating system:

NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.6 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Error message:

Error executing process > 'deepsig (2)'

Caused by:
  Process `deepsig (2)` terminated with an error exit status (1)

Command executed:

  CHUNKSIZE="$(decide_task_chunksize.sh in.fasta "4" 100)"

  run () {
      set -e
      OUT="tmp$$"
      deepsig.py -f $1 -k euk -o "${OUT}" 1>&2
      cat "${OUT}"
      rm -f "${OUT}"
  }

  export -f run

  parallel         --halt now,fail=1         --joblog log.txt         -j "4"         -N "${CHUNKSIZE}"         --line-buffer          --recstart '>'         --cat run     < in.fasta     | cat > out.txt

  predutils r2js         --pipeline-version "1.2.6"         --software-version "0f1e1d9"         -o out.ldjson         deepsig out.txt in.fasta

Command exit status:
  1

Command output:
  (empty)

Command error:
  2022-07-21 20:36:53.933482: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/slurm/latest/lib64
  2022-07-21 20:36:53.933517: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  2022-07-21 20:36:53.933187: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/slurm/latest/lib64
  2022-07-21 20:36:53.933227: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  2022-07-21 20:36:53.933182: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/slurm/latest/lib64
  2022-07-21 20:36:53.933218: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  2022-07-21 20:36:53.933187: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/slurm/latest/lib64
  2022-07-21 20:36:53.933236: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  Traceback (most recent call last):
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/share/deepsig-0f1e1d9-2/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
  ImportError: cannot import name 'pad_sequences' from 'keras.preprocessing.sequence' (/mnt/shared/scratch/theaven/apps/conda/lib/python3.9/site-packages/keras/preprocessing/sequence.py)
  Traceback (most recent call last):
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/share/deepsig-0f1e1d9-2/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
  ImportError: cannot import name 'pad_sequences' from 'keras.preprocessing.sequence' (/mnt/shared/scratch/theaven/apps/conda/lib/python3.9/site-packages/keras/preprocessing/sequence.py)
  Traceback (most recent call last):
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/share/deepsig-0f1e1d9-2/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
  ImportError: cannot import name 'pad_sequences' from 'keras.preprocessing.sequence' (/mnt/shared/scratch/theaven/apps/conda/lib/python3.9/site-packages/keras/preprocessing/sequence.py)
  Traceback (most recent call last):
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/bin/deepsig.py", line 15, in <module>
      from deepsiglib.helpers import printDate
    File "/mnt/shared/scratch/theaven/apps/conda/envs/predector/share/deepsig-0f1e1d9-2/deepsiglib/helpers.py", line 13, in <module>
      from keras.preprocessing.sequence import pad_sequences
  ImportError: cannot import name 'pad_sequences' from 'keras.preprocessing.sequence' (/mnt/shared/scratch/theaven/apps/conda/lib/python3.9/site-packages/keras/preprocessing/sequence.py)
  parallel: This job failed:
  run $PARALLEL_TMP

conda list:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_py-xgboost-mutex         2.0                       cpu_0    conda-forge
_r-mutex                  1.0.1               anacondar_1    conda-forge
abseil-cpp                20210324.2           h9c3ff4c_0    conda-forge
absl-py                   1.1.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.8.1            py39hb9d737c_1    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.6.1              h7f98852_0    conda-forge
apoplastp                 1.0.1                         1    predector
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
binutils_impl_linux-64    2.36.1               h193b22a_2    conda-forge
binutils_linux-64         2.36                hf3e587d_10    conda-forge
bioconductor-biocgenerics 0.40.0            r41hdfd78af_0    bioconda
bioconductor-biostrings   2.62.0            r41h5c21468_1    bioconda
bioconductor-genomeinfodb 1.30.0            r41hdfd78af_0    bioconda
bioconductor-genomeinfodbdata 1.2.7             r41hdfd78af_1    bioconda
bioconductor-iranges      2.28.0            r41h5c21468_1    bioconda
bioconductor-s4vectors    0.32.3            r41h5c21468_0    bioconda
bioconductor-xvector      0.34.0            r41h5c21468_1    bioconda
bioconductor-zlibbioc     1.40.0            r41h5c21468_1    bioconda
biopython                 1.79             py39hb9d737c_2    conda-forge
blinker                   1.4                        py_1    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py39hb9d737c_1004    conda-forge
bwidget                   1.9.14               ha770c72_1    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.6.15            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            ha12eb4b_1010    conda-forge
certifi                   2022.6.15        py39hf3d152e_0    conda-forge
cffi                      1.15.1           py39he91dace_0    conda-forge
charset-normalizer        2.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3            py39hf3d152e_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
coreutils                 9.1                  h166bdaf_0    conda-forge
cryptography              37.0.4           py39hd97740a_0    conda-forge
curl                      7.83.1               h7bff187_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
deeploc                   1.0                      pypi_0    pypi
deepsig                   0f1e1d9                       2    predector
effectorp1                1.0                           1    predector
effectorp2                2.0                           1    predector
effectorp3                3.0                        py_2    predector
emboss                    6.6.0                h1b6f16a_5    bioconda
expat                     2.4.8                h27087fc_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.34.4           py39hb9d737c_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
frozenlist                1.3.0            py39hb9d737c_1    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
gawk                      5.1.0                h7f98852_0    conda-forge
gcc_impl_linux-64         10.3.0              hf2f2afa_16    conda-forge
gcc_linux-64              10.3.0              hc39de41_10    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
gfortran_impl_linux-64    10.3.0              h73f4979_16    conda-forge
gfortran_linux-64         10.3.0              hb09a455_10    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glib                      2.72.1               h6239696_0    conda-forge
glib-tools                2.72.1               h6239696_0    conda-forge
google-auth               1.35.0             pyh6c4a22f_0    conda-forge
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpc-cpp                  1.42.0               ha1441d3_1    conda-forge
grpcio                    1.42.0           py39hff7568b_0    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
gst-plugins-base          1.20.3               hf6a322e_0    conda-forge
gstreamer                 1.20.3               hd4edc92_0    conda-forge
gxx_impl_linux-64         10.3.0              hf2f2afa_16    conda-forge
gxx_linux-64              10.3.0              h2593f52_10    conda-forge
gzip                      1.12                 h166bdaf_0    conda-forge
h5py                      3.7.0           nompi_py39h63b1161_100    conda-forge
harfbuzz                  4.2.0                h40b6f09_0    conda-forge
hdf5                      1.12.1          nompi_h2386368_104    conda-forge
hmmer                     3.3.2                h87f3376_2    bioconda
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4           py39hf3d152e_0    conda-forge
importlib-resources       5.8.0                    pypi_0    pypi
intel-openmp              2022.0.1          h06a4308_3633
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
keras                     2.7.0              pyhd8ed1ab_0    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_15    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py39hf939315_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lasagne                   0.2.dev1           pyh39e3cac_0    predector
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0            14_linux64_mkl    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0            14_linux64_mkl    conda-forge
libclang                  13.0.1          default_hc23dcda_0    conda-forge
libcurl                   7.83.1               h7bff187_0    conda-forge
libdeflate                1.12                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-devel_linux-64     10.3.0              he6cfe16_16    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgd                     2.3.3                h283352f_2    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libglib                   2.72.1               h2d90d5f_0    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
libgpuarray               0.7.6             h7f98852_1003    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libidn2                   2.3.3                h166bdaf_0    conda-forge
liblapack                 3.9.0            14_linux64_mkl    conda-forge
libllvm13                 13.0.1               hf817b99_2    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h753d276_3    conda-forge
libpq                     14.4                 hd77ab85_0    conda-forge
libprotobuf               3.19.4               h780b84a_0    conda-forge
libsanitizer              10.3.0              h26c7422_16    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-devel_linux-64  10.3.0              he6cfe16_16    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.4.0                hc85c160_1    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.2.3                h522a892_0    conda-forge
libwebp-base              1.2.3                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxgboost                1.5.1            cpu_h3d145d1_2    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzlib                   1.2.12               h166bdaf_2    conda-forge
llvm-meta                 7.0.0                         0    conda-forge
localizer                 1.0.4                         0    predector
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
make                      4.3                  hd18ef5c_1    conda-forge
mako                      1.2.1              pyhd8ed1ab_0    conda-forge
markdown                  3.4.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py39hb9d737c_1    conda-forge
matplotlib                3.5.2            py39hf3d152e_0    conda-forge
matplotlib-base           3.5.2            py39h700656a_0    conda-forge
mkl                       2022.0.1           h06a4308_117
mmseqs2                   13.45111             h95f258a_1    bioconda
multidict                 6.0.2            py39hb9d737c_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.29               haf5c9bc_1    conda-forge
mysql-libs                8.0.29               h28c427c_1    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nextflow                  22.04.0              h4a94de4_0    bioconda
ninja                     1.11.0               h924138e_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.78                 h2350873_0    conda-forge
numpy                     1.23.1           py39hba7629e_0    conda-forge
oauthlib                  3.2.0              pyhd8ed1ab_0    conda-forge
openjdk                   11.0.1            h516909a_1016    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openmp                    7.0.0                h2d50403_0    conda-forge
openssl                   1.1.1q               h166bdaf_0    conda-forge
opt_einsum                3.3.0              pyhd8ed1ab_1    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.3            py39h1832856_0    conda-forge
pango                     1.50.7               hbd2fdc8_0    conda-forge
parallel                  20220522             ha770c72_0    conda-forge
patch                     2.7.6             h7f98852_1002    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pcre2                     10.37                h032f7d1_0    conda-forge
perl                      5.26.2            h36c2ea0_1008    conda-forge
perl-apache-test          1.40                    pl526_1    bioconda
perl-app-cpanminus        1.7044                  pl526_1    bioconda
perl-base                 2.23                    pl526_1    bioconda
perl-bioperl              1.6.924                       4    bioconda
perl-carp                 1.38                    pl526_3    bioconda
perl-class-load           0.25                    pl526_0    bioconda
perl-class-load-xs        0.10            pl526h6bb024c_2    bioconda
perl-constant             1.33                    pl526_1    bioconda
perl-data-optlist         0.110                   pl526_2    bioconda
perl-devel-globaldestruction 0.14                    pl526_0    bioconda
perl-devel-overloadinfo   0.005                   pl526_0    bioconda
perl-devel-stacktrace     2.04                    pl526_0    bioconda
perl-dist-checkconflicts  0.11                    pl526_2    bioconda
perl-eval-closure         0.14            pl526h6bb024c_4    bioconda
perl-exporter             5.72                    pl526_1    bioconda
perl-extutils-makemaker   7.36                    pl526_1    bioconda
perl-getopt-long          2.50                    pl526_1    bioconda
perl-io-tty               1.12                    pl526_1    bioconda
perl-ipc-run              20180523.0              pl526_0    bioconda
perl-module-implementation 0.09                    pl526_2    bioconda
perl-module-runtime       0.016                   pl526_1    bioconda
perl-module-runtime-conflicts 0.003                   pl526_0    bioconda
perl-moose                2.2011          pl526hf484d3e_1    bioconda
perl-mro-compat           0.13                    pl526_0    bioconda
perl-package-deprecationmanager 0.17                    pl526_0    bioconda
perl-package-stash        0.38            pl526hf484d3e_1    bioconda
perl-package-stash-xs     0.28            pl526hf484d3e_1    bioconda
perl-params-util          1.07            pl526h6bb024c_4    bioconda
perl-parent               0.236                   pl526_1    bioconda
perl-pathtools            3.75            pl526h14c3975_1    bioconda
perl-scalar-list-utils    1.52            pl526h516909a_0    bioconda
perl-sub-exporter         0.987                   pl526_2    bioconda
perl-sub-exporter-progressive 0.001013                pl526_0    bioconda
perl-sub-identify         0.14            pl526h14c3975_0    bioconda
perl-sub-install          0.928                   pl526_2    bioconda
perl-sub-name             0.21                    pl526_1    bioconda
perl-threaded             5.32.1               hdfd78af_1    bioconda
perl-try-tiny             0.30                    pl526_1    bioconda
perl-xsloader             0.24                    pl526_0    bioconda
perl-yaml                 1.29                    pl526_0    bioconda
pfam_scan                 1.6                  hdfd78af_4    bioconda
phobius                   1.01                          4    predector
pillow                    9.2.0            py39hae2aec6_0    conda-forge
pip                       22.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
predectorutils            0.8.3              pyh7b7c402_0    predector
protobuf                  3.19.4           py39he80948d_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py-xgboost                1.5.1           cpu_py39h4655687_2    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygpu                     0.7.6           py39hce5d2b2_1003    conda-forge
pyjwt                     2.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.12.3           py39h03dd644_4    conda-forge
pyqt5-sip                 4.19.18                  pypi_0    pypi
pyqtchart                 5.12                     pypi_0    pypi
pyqtwebengine             5.12.1                   pypi_0    pypi
pysocks                   1.7.1            py39hf3d152e_5    conda-forge
python                    3.9.13          h9a8a25e_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        2.0                pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytorch                   1.11.0          cpu_py39h7613f69_0    conda-forge
pytorch-cpu               1.11.0          cpu_py39h718b53a_0    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
qt                        5.12.9               h1304e3e_6    conda-forge
r-ade4                    1.7_19            r41h0154571_0    conda-forge
r-assertthat              0.2.1             r41hc72bb7e_2    conda-forge
r-backports               1.4.1             r41hcfec24a_0    conda-forge
r-base                    4.1.3                hd930d0e_0    conda-forge
r-base64enc               0.1_3           r41hcfec24a_1004    conda-forge
r-bitops                  1.0_7             r41h06615bd_0    conda-forge
r-brio                    1.1.3             r41hcfec24a_0    conda-forge
r-callr                   3.7.1             r41hc72bb7e_0    conda-forge
r-cli                     3.3.0             r41h7525677_0    conda-forge
r-colorspace              2.0_3             r41h06615bd_0    conda-forge
r-config                  0.3.1             r41hc72bb7e_0    conda-forge
r-crayon                  1.5.1             r41hc72bb7e_0    conda-forge
r-deepredeff              0.1.1             r41h142f84f_0    predector
r-desc                    1.4.1             r41hc72bb7e_0    conda-forge
r-diffobj                 0.3.5             r41hcfec24a_0    conda-forge
r-digest                  0.6.29            r41h03ef668_0    conda-forge
r-dplyr                   1.0.9             r41h7525677_0    conda-forge
r-ellipsis                0.3.2             r41hcfec24a_0    conda-forge
r-evaluate                0.15              r41hc72bb7e_0    conda-forge
r-fansi                   1.0.3             r41h06615bd_0    conda-forge
r-farver                  2.1.1             r41h7525677_0    conda-forge
r-fs                      1.5.2             r41h7525677_1    conda-forge
r-generics                0.1.3             r41hc72bb7e_0    conda-forge
r-getopt                  1.20.3            r41ha770c72_2    conda-forge
r-ggplot2                 3.3.6             r41hc72bb7e_0    conda-forge
r-ggthemes                4.2.4             r41hc72bb7e_0    conda-forge
r-glue                    1.6.2             r41h06615bd_0    conda-forge
r-gtable                  0.3.0             r41hc72bb7e_3    conda-forge
r-here                    1.0.1             r41hc72bb7e_0    conda-forge
r-hms                     1.1.1             r41hc72bb7e_0    conda-forge
r-isoband                 0.2.5             r41h03ef668_0    conda-forge
r-jsonlite                1.8.0             r41h06615bd_0    conda-forge
r-keras                   2.9.0             r41hc72bb7e_0    conda-forge
r-labeling                0.4.2             r41hc72bb7e_1    conda-forge
r-lattice                 0.20_45           r41hcfec24a_0    conda-forge
r-lifecycle               1.0.1             r41hc72bb7e_0    conda-forge
r-magrittr                2.0.3             r41h06615bd_0    conda-forge
r-mass                    7.3_58            r41h06615bd_0    conda-forge
r-matrix                  1.4_1             r41h0154571_0    conda-forge
r-mgcv                    1.8_40            r41h0154571_0    conda-forge
r-munsell                 0.5.0           r41hc72bb7e_1004    conda-forge
r-nlme                    3.1_158           r41h8da6f51_0    conda-forge
r-optparse                1.7.3             r41hc72bb7e_0    conda-forge
r-pillar                  1.8.0             r41hc72bb7e_0    conda-forge
r-pixmap                  0.4_12            r41hc72bb7e_0    conda-forge
r-pkgconfig               2.0.3             r41hc72bb7e_1    conda-forge
r-pkgload                 1.3.0             r41hc72bb7e_0    conda-forge
r-png                     0.1_7           r41hcfec24a_1004    conda-forge
r-praise                  1.0.0           r41hc72bb7e_1005    conda-forge
r-prettyunits             1.1.1             r41hc72bb7e_1    conda-forge
r-processx                3.7.0             r41h06615bd_0    conda-forge
r-progress                1.2.2             r41hc72bb7e_2    conda-forge
r-ps                      1.7.1             r41h06615bd_0    conda-forge
r-purrr                   0.3.4             r41hcfec24a_1    conda-forge
r-r6                      2.5.1             r41hc72bb7e_0    conda-forge
r-rappdirs                0.3.3             r41hcfec24a_0    conda-forge
r-rcolorbrewer            1.1_3             r41h785f33e_0    conda-forge
r-rcpp                    1.0.9             r41h7525677_0    conda-forge
r-rcpptoml                0.1.7             r41h03ef668_1    conda-forge
r-rcurl                   1.98_1.7          r41h06615bd_0    conda-forge
r-rematch2                2.1.2             r41hc72bb7e_1    conda-forge
r-reticulate              1.25              r41h7525677_1    conda-forge
r-rlang                   1.0.4             r41h7525677_0    conda-forge
r-rprojroot               2.0.3             r41hc72bb7e_0    conda-forge
r-rstudioapi              0.13              r41hc72bb7e_0    conda-forge
r-scales                  1.2.0             r41hc72bb7e_0    conda-forge
r-segmented               1.5_0             r41hc72bb7e_0    conda-forge
r-seqinr                  4.2_16            r41h06615bd_0    conda-forge
r-sp                      1.5_0             r41h06615bd_0    conda-forge
r-stringi                 1.7.6             r41h337692f_1    conda-forge
r-stringr                 1.4.0             r41hc72bb7e_2    conda-forge
r-tensorflow              2.9.0             r41hc72bb7e_0    conda-forge
r-testthat                3.1.4             r41h7525677_0    conda-forge
r-tfautograph             0.3.2             r41hc72bb7e_1    conda-forge
r-tfruns                  1.5.0             r41hc72bb7e_0    conda-forge
r-tibble                  3.1.7             r41h06615bd_0    conda-forge
r-tidyselect              1.1.2             r41hc72bb7e_0    conda-forge
r-utf8                    1.2.2             r41hcfec24a_0    conda-forge
r-vctrs                   0.4.1             r41h7525677_0    conda-forge
r-viridislite             0.4.0             r41hc72bb7e_0    conda-forge
r-waldo                   0.4.0             r41hc72bb7e_0    conda-forge
r-whisker                 0.4               r41hc72bb7e_1    conda-forge
r-withr                   2.5.0             r41hc72bb7e_0    conda-forge
r-yaml                    2.3.5             r41h06615bd_0    conda-forge
r-zeallot                 0.1.0           r41hc72bb7e_1003    conda-forge
re2                       2021.11.01           h9c3ff4c_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scikit-learn              1.1.1            py39h4037b75_0    conda-forge
scipy                     1.8.1            py39he49c0e8_0    conda-forge
sed                       4.8                  he412f7d_0    conda-forge
setuptools                63.2.0           py39hf3d152e_0    conda-forge
signalp3                  3.0b                          3    predector
signalp4                  4.1g                          3    predector
signalp5                  5.0b                          2    predector
signalp6                  6.0+g                    pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
snappy                    1.1.9                hbd366e4_1    conda-forge
sqlite                    3.39.1               h4ff8645_0    conda-forge
sysroot_linux-64          2.12                he073ed8_15    conda-forge
targetp2                  2.0                           2    predector
tensorboard               2.6.0              pyhd8ed1ab_1    conda-forge
tensorboard-data-server   0.6.0            py39hd97740a_2    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.7.0           cpu_py39h4655687_0    conda-forge
tensorflow-base           2.7.0           cpu_py39hf4995fd_0    conda-forge
tensorflow-estimator      2.7.0           cpu_py39ha241409_0    conda-forge
termcolor                 1.1.0              pyhd8ed1ab_3    conda-forge
theano                    1.0.5            py39h5a03fae_3    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tktable                   2.10                 hb7b940f_3    conda-forge
tmhmm                     2.0c                          3    predector
tornado                   6.2              py39hb9d737c_0    conda-forge
tqdm                      4.64.0             pyhd8ed1ab_0    conda-forge
typing-extensions         4.3.0                hd8ed1ab_0    conda-forge
typing_extensions         4.3.0              pyha770c72_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
unicodedata2              14.0.0           py39hb9d737c_1    conda-forge
unzip                     6.0                  h7f98852_3    conda-forge
urllib3                   1.26.10            pyhd8ed1ab_0    conda-forge
werkzeug                  2.1.2              pyhd8ed1ab_1    conda-forge
wget                      1.20.3               ha56f1ee_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py39hb9d737c_0    conda-forge
xgboost                   1.5.1           cpu_py39h4655687_2    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-libxt                1.2.1                h7f98852_2    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yarl                      1.7.2            py39hb9d737c_2    conda-forge
zip                       3.0                  h7f98852_1    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               h166bdaf_2    conda-forge
zstd                      1.5.2                h8a70e8d_2    conda-forge
TCHeaven commented 2 years ago

I have discovered that I can avoid the above error if I specify predector to run with certain nodes on my HPC, it is unclear to me what differentiates those nodes that trigger the error from those that do not.

No deepsig error with: CPU GPU Memory Disk 2x Silver 4216 @2.1Ghz 16C/32T NA 192 GB 1.3 TB (SSD) 4x Gold 6242 @2.8Ghz 16C/32T 2x V100 32GB 3.0 TB 1.3 TB (SSD) 2x E5-2680 v4 @2.4GHz 14C/24T NA 1.5 TB 370 GB (SSD)

However, I now receive errors with localizer;

Error executing process > 'localizer (1)'

Caused by:
  Process `localizer (1)` terminated with an error exit status (1)

Command executed:

  run () {
      set -e
      TMP="tmp$$"
      LOCALIZER.py -e -i "$1" -o "${TMP}" 1>&2
      cat "${TMP}/Results.txt"

      rm -rf -- "${TMP}"
  }
  export -f run

  CHUNKSIZE="$(decide_task_chunksize.sh in.fasta "16" 100)"

  parallel         --halt now,fail=1         --joblog log.txt         -j "16"         -N "${CHUNKSIZE}"         --line-buffer          --recstart '>'         --cat          run     < in.fasta     | cat > out.txt

  predutils r2js         --pipeline-version "1.2.6"         --software-version "1.0.3"         -o out.ldjson         localizer out.txt in.fasta

Command exit status:
  1

Command output:
  (empty)

Command error:
  Error calling pepstats: [Errno 14] Bad address: '/mnt/shared/scratch/theaven/apps/conda/envs/predector/bin/pepstats'
  parallel: This job failed:
  run $PARALLEL_TMP
TCHeaven commented 2 years ago

I am having no luck getting past the pepstats error, pepstats’ path should be fine so far as I can tell. I do not have permissions to build singularity images on my hpc and when I try to build one locally installation of predector fails, I believe because my pc does not have an nvidia gpu.

darcyabjones commented 2 years ago

Hey @TCHeaven Sorry i've been so bad at keeping up with these.

I'm unsure what's happening with the keras issue. The only reason I can imagine is if some nodes already a different version of the keras libraries installed and they've set a default PYTHONPATH that causes python to find that library first. But really it's just a guess and I don't think it's something I can control, sounds like something weird in your compute environment. But good work figuring out which nodes work.

RE the pepstats issue. So the error is caused by the pepstats binary trying to run code from invalid memory addresses. See https://utcc.utoronto.ca/~cks/space/blog/linux/KernelSegfaultErrorCodes#:~:text=error%2014%20%3A%20attempt%20to%20execute,return%20address%20isn't%20valid.

I suspect that the pepstats binary is corrupted somehow, or that it's compiles with instructions that your computer doesn't support. Other than just re-building the whole environment again (or forcing conda to reinstall "emboss")the best option would be to re-compile emboss yourself on your computer and load it into the conda env. I've just run a quick test and this should do it.

conda activate predector
conda remove -n predector --force-remove -y --verbose emboss

wget ftp://emboss.open-bio.org/pub/EMBOSS/EMBOSS-6.6.0.tar.gz
tar -zxf EMBOSS-6.6.0.tar.gz
cd EMBOSS-6.6.0
./configure --prefix /mnt/shared/scratch/theaven/apps/conda/envs/predector --without-x
make  # this will take a while
make install

This should install a locally compiled version of emboss (which includes pepstats) into your conda environment. Hopefully that would solve your error with emboss.

RE issues with singularity. I'm happy to help with this too if you can provide details of the error :) None of the dependencies require a GPU to install or run (they won't even use one if you have one) so it'll be some other issue, but i'd need to the the details.

darcyabjones commented 2 years ago

I'd be happy to DM you a download to a singularity image if you'd rather. Just send me your email address and i'll forward the link. I wouldn't want to distribute it widely but for the sake of getting you going i feel like I could justify pushing the boundaries of the software licenses.

darcyabjones commented 2 years ago

We resolved this via email. Thanks TCHeaven :)