search

gagneurlab / MMSplice_MTSplice

Tissue-specific variant effect predictions on splicing
MIT License
40 stars 21 forks source link

FAILED to annotate in VEP #28

Closed loganylchen closed 5 years ago

loganylchen commented 5 years ago

Description

I install mmsplice based on the ensemblorg/ensembl-vep image, and with following commands

apt-get update && apt-get install -y software-properties-common && \
    add-apt-repository -y ppa:jonathonf/python-3.6  && apt-get update \
    python3.6 \
    python3.6-dev

git clone git://github.com/gagneurlab/MMSplice.git && \
  cd MMSplice && git checkout v0.2.7 && cd .. && \
  cp MMSplice/VEP_plugin/MMSplice.pm ./Plugins/

and put ./Plugins into the PERL5LIB environment.

And try to use vep to do the annotation as the following command.

What I Did

vep --format vcf --cache --offline --force_overwrite --fork 20 --sift b --polyphen b --numbers --biotype --total_length --canonical --ccds --hgvs   -q --refseq --offline --vcf --af_1kg --af --pubmed --plugin MMSplice --af_gnomad --dir_cache {database}   --input_file variants.vcf --output_file vep.vcf  --fasta human_g1k_v37.fasta

It has following INFO:

Using TensorFlow backend.
WARNING: Logging before flag parsing goes to stderr.
W0816 02:18:26.399252 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/__init__.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=DeprecationWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator HuberRegressor from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator StandardScaler from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator LogisticRegression from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/base.py:306: UserWarning: Trying to unpickle estimator Pipeline from version 0.19.2 when using version 0.21.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
W0816 02:18:26.448383 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
W0816 02:18:26.448706 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
W0816 02:18:26.453226 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:102: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
W0816 02:18:26.458684 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
W0816 02:18:26.462359 140276516513600 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
2019-08-16 02:18:26.540076: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-08-16 02:18:26.637516: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2095085000 Hz
2019-08-16 02:18:26.638957: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4731600 executing computations on platform Host. Devices:
2019-08-16 02:18:26.638996: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2019-08-16 02:18:26.768802: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '
W0816 02:18:27.275322 140276516513600 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
W0816 02:18:27.928400 140276516513600 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '
Traceback (most recent call last):
Uncaught exception from user code:

    -------------------- EXCEPTION --------------------
    MSG:
    ERROR: Forked process(es) died: read-through of cross-process communication detected

    STACK Bio::EnsEMBL::VEP::Runner::_forked_buffer_to_output /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm:556
    STACK Bio::EnsEMBL::VEP::Runner::next_output_line /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm:361
    STACK Bio::EnsEMBL::VEP::Runner::run /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm:202
    STACK toplevel /opt/vep/src/ensembl-vep/vep:223
    Date (localtime)    = Fri Aug 16 02:18:52 2019
    Ensembl API version = 97
    ---------------------------------------------------
    Bio::EnsEMBL::Utils::Exception::throw("\x{a}ERROR: Forked process(es) died: read-through of cross-proces"...) called at /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm line 556
    Bio::EnsEMBL::VEP::Runner::_forked_buffer_to_output(Bio::EnsEMBL::VEP::Runner=HASH(0x5615ec5ef8a0), Bio::EnsEMBL::VEP::InputBuffer=HASH(0x5615ed5e4cc8), undef) called at /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm line 361
    Bio::EnsEMBL::VEP::Runner::next_output_line(Bio::EnsEMBL::VEP::Runner=HASH(0x5615ec5ef8a0)) called at /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm line 202
    Bio::EnsEMBL::VEP::Runner::run(Bio::EnsEMBL::VEP::Runner=HASH(0x5615ec5ef8a0)) called at /opt/vep/src/ensembl-vep/vep line 223

  File "/usr/local/bin/mmsplice", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mmsplice/main.py", line 37, in run
    variant = json.loads(sys.stdin.readline().strip())
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 8195 (char 8194)
s6juncheng commented 5 years ago

Hi @btrspg, can you please try with the latest MMSplice version (1.0.1) and the latest vep plugin from the master branch? MMSplice version 0.2.7 does not work with the latest VEP plugin. Besides, you need to install MMSplice pip install mmsplice will get you the latest 1.0.1 version.

loganylchen commented 5 years ago

@s6juncheng Thanks, I have tried the dockerfile in the rep to build the image, it came out the same error.

MuhammedHasan commented 5 years ago

@btrspg

Can you try to run the following code to test mmsplice VEP plugin before analyzing your data:

git clone git://github.com/gagneurlab/MMSplice.git
docker build -t mmsplice .
cat tests/data/test.vcf | sudo docker run -i mmsplice vep --plugin MMSplice --format vcf --assembly GRCh37 --database --port 3337 --vcf -o STDOUT | tee variant_effect_output.txt

This works in my environment and CircleCI so should work in your environment too. The problem you counter is probably due to the parameters you run the VEP plugin. Especially --fork=20 because MMSplice VEP plugin is not thread-safe.

Thanks for reporting the issue. I have updated the documentation for thread-safety. https://github.com/gagneurlab/MMSplice/commit/7f4aeb8bfa6cd460bccc5db593c066d1691bf1f6