Stalls out part way through

liliblu commented 3 years ago

Hello, I am having an issue using deephlapan. I used the docker installation on MacOS. When I run the program on a test peptide or a file of several peptides, it stop at the same step. First, I get several copies of the same warnings:

WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead

Then a progress bar is shown:

100/100 [==============================] - 2s     
100/100 [==============================] - 3s     
100/100 [==============================] - 3s     
100/100 [==============================] - 3s     
100/100 [==============================] - 4s     
100/100 [==============================] - 1s

Then the program does not progress or fail.

If I stop the program, it says it stopped here:

Process PoolWorker-531:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    racquire()
KeyboardInterrupt

I can provide more information if needed. Any guidance would be appreciated. Thank you!

wujingcheng commented 3 years ago

Thanks for your question. We have just checked the webserver and docker of DeepHLApan in Windows and Linux. The tool works fine. I wonder if the system induces the problem.

wujingcheng commented 3 years ago

Could you provide the file that you submitted to the webserver?

rhalperin commented 3 years ago

I am experience the same issue. I am also running the docker image on MacOS (Catalina). I am just trying to run the example deephlapan -P LNIMNKLNI -H HLA-A02:01. While I'm running it I watched docker stat. There are a few minutes of heavy cpu usage, then cpu usage drops to <1%, and it just hangs. My full stdout is below. Please let me know if you need any additional information to troubleshoot. Thanks!

root@e0be51bee6b5:/notebooks# deephlapan -P LNIMNKLNI -H HLA-A02:01 Using TensorFlow backend.

    =====================================================================
    DeepHLApan is a deep learning approach used for predicting high-confidence 
    neoantigens by considering both the presentation possibilities of 
    mutant peptides and the potential immunogenicity of pMHC.

    Usage:

    Single peptide and HLA:

        deephlapan -P LNIMNKLNI -H HLA-A02:01 

    List of peptides and HLA alleles in a file:

        deephlapan -F [file] -O [output directory]  

        (see 1.csv in demo/ for the detailed format of input file)
    =====================================================================

2021-01-28 17:02:42.422698 Prediction starting.....

WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1188: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 2021-01-28 17:02:45.924818: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 2021-01-28 17:02:46.416046: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-01-28 17:02:46.437215: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-01-28 17:02:46.464923: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-01-28 17:02:46.552010: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-01-28 17:02:46.622187: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s 1/1 [==============================] - 1s

depillis commented 2 years ago

I am having a similar issue: the code runs then hangs unless I have 5 or fewer entries in the CSV file.

In addition, when the code does complete, it generates a "rank" file that is empty.

I tested this on my own csv files and on the 1.csv file that comes in the demo directory.

Here are the final lines printed (there are 299 entries in my smaller csv file): WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 299/299 [==============================] - 5s 299/299 [==============================] - 5s 299/299 [==============================] - 6s 299/299 [==============================] - 6s 299/299 [==============================] - 7s 299/299 [==============================] - 7s 299/299 [==============================] - 2s 299/299 [==============================] - 2s 288/299 [===========================>..] - ETA: 0s

Tried it on progessively smaller files: 100 entries, 50, 15, 10, and finally only 5 entries.

At 10 entries it hangs: packages/Keras-2.0.8-py2.7.egg/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead 10/10 [==============================] - 1s 10/10 [==============================] - 1s 10/10 [==============================] - 1s 10/10 [==============================] - 1s 10/10 [==============================] - 9s 10/10 [==============================] - 9s 10/10 [==============================] - 9s

But 5 entries is able to complete. It generates two files, one called "...predicted_result_rank.csv" and "...predicted_result.csv."

My questions are:

Are there settings to adjust to help the code handle larger csv files?
Why is the "rank" file empty? Are there settings to address that?

I'm running this inside a Docker container on Mac OS Catalina.

depillis commented 2 years ago

A follow-up to my message from 6 days ago: I've tried the CSV file by submitting to the DeepHLApan server, and it seems to stall there as well. I receive an email stating "biopharm | | 12:35 PM (6 minutes ago) Your job of MHCI_Allele_binding_LdePModified.csv is finished!" but (1) I cannot see results anywhere, and (2) on the page submission page http://biopharm.zju.edu.cn/deephlapan/submit/ there is still a message stating "please waiting" with a spinning circle. So it seems to be hanging on the server side as well.

The same thing happens with the test file provided, 1.csv.

jiujiezz / deephlapan

Stalls out part way through #26