search

haotianteng / Chiron

A basecaller for Oxford Nanopore Technologies' sequencers
Other
122 stars 53 forks source link

Deprecated tenslorflow #83

Closed tarunaaggarwal closed 5 years ago

tarunaaggarwal commented 5 years ago

Hi @haotianteng - I followed the pip install instructions you have posted in the README and tried to basecall using the default model and got a bunch of errors. Below is the info you will need to help resolve this issue.

Command chiron call -i /scratch/taruna/20190226_2329_MN28003_FAJ01459_89dd5206 -o /scratch/taruna/chiron-output

Error

'model_default_path', '/home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/model/DNA_default')
Subdirectory processing:: 2it [00:00, 136.27it/s]
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/rnn.py:49: __init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/rnn.py:61: __init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/rnn.py:65: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/chiron_eval.py:384: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/chiron/chiron_eval.py:385: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
2019-04-08 23:11:24.099277: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-04-08 23:11:24.651452: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2019-04-08 23:11:24.658721: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55cebb341740 executing computations on platform Host. Devices:
2019-04-08 23:11:24.658815: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/taruna/lib/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.

/scratch/taruna/chiron-output
Real time:4.653 Systime:0.190 Usertime:1.662

Thanks

pdimens commented 5 years ago

I'm having this exact same issue. The progress bar for "Scanning Signal" parses through to completion, then the long deprecation text. Have tried on both python3.7 and python2.7 with tensorflow1.0.1 and current.

> chiron call -i nanopore_YFT -o basecalled

model_default_path /usr/lib/python3.7/site-packages/chiron/model/DNA_default
Subdirectory processing:: 1it [00:00,  2.10it/s]
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/chiron/rnn.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/chiron/rnn.py:61: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/chiron/rnn.py:65: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/chiron/chiron_eval.py:384: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/chiron/chiron_eval.py:385: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
2019-04-09 17:54:27.155356: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-09 17:54:27.187573: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3094235000 Hz
2019-04-09 17:54:27.191078: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561126ca3100 executing computations on platform Host. Devices:
2019-04-09 17:54:27.191123: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /usr/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.

basecalled
Real time:1.287 Systime:0.134 Usertime:1.251
pdimens commented 5 years ago

The manager of our Uni's cluster took a stab at it and successfully got it running. The instructions they provided me worked on my system as well, so here are the explicit instructions:

virtualenv -p python3.7 venv/chiron
source venv/chiron/bin/activate
pip install chiron
pip install tensorflow
ln -s ~/venv/chiron/lib/python3.7/site-packages/chiron .
mkdir output
python chiron/entry.py call -i chiron/example_data -o output/ -m chiron/model/DNA_default -b 1100 -l 400 -j 40 --beam 30 --mode dna
tarunaaggarwal commented 5 years ago

Hi @pdimens - thank you so much for sharing your solution. I will definitely try it out today. Hopefully, it works for me too. Thank you!

Taruna

pdimens commented 5 years ago

It worked for me quite easily, so I hope it will for you as well. I think my initial issue was trying to have a conda environment (as that's what I'm used to), but also having pip installation, but my systems are also Arch linux, which prefers you install python packages via their repositories-- and it was messing things up.

I should say fully credit goes to Brian Olson at the University of Southern Mississippi because I reached out to him when I was all out of guesses.

tarunaaggarwal commented 5 years ago

I see. I was also creating a conda environment. Did Brian tell you why creating a conda environment was causing issues? Was it just bc of your system and it's finickiness?

pdimens commented 5 years ago

He didn't and I didn't ask, haha. Normally I would pursue the question, but at the moment I'm glad it can run and my work can proceed. Perhaps I'll revisit this later when the dust settles a bit. I'm sure there's a sensible way to do it in conda that I overlooked.

tarunaaggarwal commented 5 years ago

Haha. Yes, it would be nice to know. May be @haotianteng will have some input. Thank you again!

pdimens commented 5 years ago

Strange, it works on example data, but won't work on my raw fast5 files.

pdimens commented 5 years ago

UPDATE: new errors (see bottom 2 lines)

python chiron/entry.py call -i /mnt/tertiary/nanopore_YFT -o basecalled -m chiron/model/DNA_default -b 1100 -l 400 -j 40 --beam 30 --mode dna -e fastq
['call', '-i', '/mnt/tertiary/nanopore_YFT', '-o', 'basecalled', '-m', 'chiron/model/DNA_default', '-b', '1100', '-l', '400', '-j', '40', '--beam', '30', '--mode', 'dna', '-e', 'fastq']
model_default_path /home/pdimens/chiron/model/DNA_default
Subdirectory processing:: 1it [00:00,  2.13it/s]
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:61: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:61: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:65: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/rnn.py:65: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/chiron_eval.py:384: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/chiron_eval.py:384: QueueRunner.__init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/chiron_eval.py:385: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/chiron/chiron_eval.py:385: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Graph was finalized.
2019-04-10 13:09:51.497761: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-04-10 13:09:51.527414: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3094235000 Hz
2019-04-10 13:09:51.530681: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561a877e36c0 executing computations on platform Host. Devices:
2019-04-10 13:09:51.530713: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py:809: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the `tf.data` module.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From /home/pdimens/venv/chiron/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from chiron/model/DNA_default/model.ckpt-209071
INFO:tensorflow:Restoring parameters from chiron/model/DNA_default/model.ckpt-209071
haotianteng commented 5 years ago

Hi, it's just a warning message, won't influence the use.

pdimens commented 5 years ago

@haotianteng , Thank you for writing back. It however doesn't process any fast5 files in my input directory.

haotianteng commented 5 years ago

Chiron has only detected 1 subdirectory. So I think the input directory you give is not correct or there is some permission issue.

haotianteng commented 5 years ago

Hi @pidmens, Have you fixed the problem? Maybe you would like to try version 0.5.1, as it has a much faster DNA model with the same accuracy as the old version(in some dataset even slightly better), also I fix the threading module that causes the software corruption in Python 2.7, not sure if it would help in your case.

pdimens commented 5 years ago

Thank you for your feedback. I haven't had a chance to test it out, unfortunately, and might not be able to for a few months. I hope the sample data I provided was helpful in diagnosing this issue and issues others may be experiencing. I will certainly let you know if/when I test this out.

haotianteng commented 5 years ago

Oh, I just notice that you are the same person that ask for the enhancement to run on the new data format. Okay, it didn't fix yet, I will close this issue and let's catch up on that issue.