Closed Musketeer-D closed 5 years ago
Hi @DAVID2022 , could you please describe more about the basealling of your data and which type of modification you will study? Thank you.
We perform realtime basealling using guppy-2.0.10 on PromethION and we want to analysis both 6mA and 5mc modifications using your great softwre on our organism.
Hi @DAVID2022 , in the updated basecalling in PromethoION, there are two issues you might meet later: (1) please check whether events
are available in fast5 files, if no, basecalling with events information might needed. (2) please check whether a fast5 file contains a single read or multiple reads, if multiple reads, you might need to use the tool from "https://github.com/nanoporetech/ont_fast5_api" for converting multif-fast5 format to single-fast format.
For 5mC, after basecalling using albacore v2.3+, you can use rnn_f7_wd21_chr1to10_4
. There is only one model for 6mA.
Feel free to let me know if you have any issue to use this tool.
Thank you very much ! I will try it instantly.
We perform realtime basealling using guppy-2.0.10 with default parameters. How can I make sure is there any events information in fast5 files ? Should I use some special tools to check that?
H5dump can be used to check all information in fast5. Please note that h5dump print fast5 content to screen.
I cannot find any information about events in my fast5 file, So to get those events information ,Is there any parameter should I set on MinKNOW?
You might use case insensitive in grep. It seems events generation might be not supported durning new Nanopore machine(not sure what your Nanopore software versions). But you can use albacore with one of option value equal to “fastq,fast5” to re-baseball with events.
I still cannot find any information about events in my fast5 file using case insensitive in grep. So if I use guppy(v3.1.5) which is the latest basecaller of ONT to perform basecalling, I cannot get the events information deepmod need ?
Hi @DAVID2022 , the message shows there is no event in the fast5 file. You might use albacore v2.3+ re-basecall fast5 files.
Thank you very much for your kind help! I will try to basecalling my data with albacore. Considering the higher speed and accuracy of guppy, hope to see you great software be compatible with guppy soon!
When I try to run deepmod on my data in this way: python /public1/home/pg3152/DeepMod/bin/DeepMod.py detect --wrkBase /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/20190203-NPL0651-P1-A11-D11/1-A11-D11/reads/0.tar --Ref /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/deepmod/data/david-david-REFERENCE-1.0.fa --outFolder /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/deepmod/data --Base C --modfile /public1/home/pg3152/DeepMod/train_mod/rnn_f7_wd21_chr1to10_4/mod_train_f7_wd21_chr1to10 --FileID david_tar0 --threads 6
I got some error information :
Nanopore sequencing data analysis is resourece-intensive and time consuming. Some potential strong recommendations are below: If your reference genome is large as human genome and your Nanopore data is huge, It would be faster to run this program parallelly to speed up. You might run different input folders of your fast5 files and give different output names (--FileID) or folders (--outFolder) A good way for this is to run different chromosome individually.
Current directory: /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/deepmod/process
outLevel: 2
wrkBase: /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/20190203-NPL0651-P1-A11-D11/1-A11-D11/reads/0.tar
FileID: david_tar0
outFolder: /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/deepmod/data/
recursive: 1
files_per_thread: 1000
threads: 6
windowsize: 21
alignStr: minimap2
basecall_1d: Basecall_1D_000
basecall_2strand: BaseCalled_template
ConUnk: True
outputlayer:
Base: C
mod_cluster: 0
predDet: 1
Ref: /public1/home/pg3152/DAVID/TASK22-3gMETH/NANO19/davidDATA/deepmod/data/david-david-REFERENCE-1.0.fa
fnum: 7
hidden: 100
modfile: /public1/home/pg3152/DeepMod/train_mod/rnn_f7_wd21_chr1to10_4/mod_train_f7_wd21_chr1to10
region: [[None, None, None]]
Total files=0
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: BasicLSTMCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:42: MultiRNNCell.init (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/DeepMod/bin/scripts/myMultiBiRNN.py:47: static_bidirectional_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use keras.layers.Bidirectional(keras.layers.RNN(cell, unroll=True))
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py:1565: static_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removedin a future version.
Instructions for updating:
Please use keras.layers.RNN(cell, unroll=True)
, which is equivalent to this API
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:526: to_float (from tensorflow.python.ops.math_ops) is deprecated and willbe removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /public1/home/pg3152/anaconda2/envs/mdeepmod/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:788: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
2019-05-30 21:16:08.429202: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.429427: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.429511: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.429612: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.429648: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.429673: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500035000 Hz
2019-05-30 21:16:08.467710: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5f500 executing computations on platform Host. Devices:
2019-05-30 21:16:08.467723: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5f6a0 executing computations on platform Host. Devices:
2019-05-30 21:16:08.467739: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5fdd0 executing computations on platform Host. Devices:
2019-05-30 21:16:08.467743: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5f7b0 executing computations on platform Host. Devices:
2019-05-30 21:16:08.467752: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5f580 executing computations on platform Host. Devices:
2019-05-30 21:16:08.467758: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5649ddf5f440 executing computations on platform Host. Devices:
2019-05-30 21:16:08.478697: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0):
Can you tell me why? What should I do to solve those problem?
Hi @DAVID2022 , DeepMod cannot support .tar now. I would add this in a later version.
Thank you!I will retry my data.
Hi, If I want to try your software on the research of some other organism, which modfile should I choose? rnn_f7_wd21_chr1to10_4 or rnn_conmodC_P100wd21_f7ne1u0_4? Best wishes, David.