PengNi
commented
1 year ago @Arkadiy-Garber , hi, just use the prefix /path/to/bn_17.sn_360.epoch_9.ckpt.
Open Arkadiy-Garber opened 1 year ago
PengNi
commented
1 year ago @Arkadiy-Garber , hi, just use the prefix /path/to/bn_17.sn_360.epoch_9.ckpt.
Arkadiy-Garber
commented
1 year ago Thanks, that seemed to worked to fix the previous issue. However, I am not getting the following output at then end of the stdout (with the command):
deepsignal call_mods --input_path single_barcode12/ --model_path /data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt --result_file single_barcode12.gatc.call_mods.tsv --corrected_group RawGenomeCorrected_001 --nproc 10 --is_gpu no
/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/sklearn/utils/validation.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
LARGE_SPARSE_SUPPORTED = LooseVersion(scipy_version) >= '0.14.0'
# ===============================================
## parameters:
input_path:
single_barcode12/
f5_batch_num:
50
model_path:
/data/MAB/bin/deepsignal/models/model.GATC.R9_2D.tem.puc19.bn17.sn360/bn_17.sn_360.epoch_5.ckpt
is_cnn:
yes
is_rnn:
yes
is_base:
yes
kmer_len:
17
cent_signals_len:
360
batch_size:
512
learning_rate:
0.001
class_num:
2
result_file:
single_barcode12.gatc.call_mods.tsv
recursively:
yes
corrected_group:
RawGenomeCorrected_001
basecall_subgroup:
BaseCalled_template
is_dna:
yes
normalize_method:
mad
motifs:
CG
mod_loc:
0
positions:
None
reference_path:
None
nproc:
1
is_gpu:
no
# ===============================================
110340 fast5 files in total..
parse the motifs string..
read genome reference file..
read position file if it is not None..
call_mods process 1352721 starts
write process 1352722 starts
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:49: LSTMCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.LSTMCell, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:66: MultiRNNCell.__init__ (from tensorflow.python.ops.rnn_cell_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This class is equivalent as tf.keras.layers.StackedRNNCells, and will be replaced by that in Tensorflow 2.0.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:71: bidirectional_dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.Bidirectional(keras.layers.RNN(cell))`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:443: dynamic_rnn (from tensorflow.python.ops.rnn) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `keras.layers.RNN(cell)`, which is equivalent to this API
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn.py:626: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/rnn_cell_impl.py:1259: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:185: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:191: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:235: average_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.average_pooling2d instead.
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/layers.py:77: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.
2023-03-24 19:35:20.760064: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2023-03-24 19:35:20.786036: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2023-03-24 19:35:20.790056: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55c23ba56f60 executing computations on platform Host. Devices:
2023-03-24 19:35:20.790083: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2023-03-24 19:35:23.436685: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
Process Process-1:
Traceback (most recent call last):
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]
Caused by op 'save/RestoreV2', defined at:
File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
args.func(args)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
f5_args)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
is_rnn, is_base, is_cnn)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
p.start()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
saver = tf.train.Saver()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
self.build()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
build_save=build_save, build_restore=build_restore)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
restore_sequentially, reshape)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
restore_sequentially)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
name=name)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
NotFoundError (see above for traceback): Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1591, in object_graph_key_mapping
checkpointable.OBJECT_GRAPH_PROTO_KEY)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 370, in get_tensor
status)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 255, in _fast5s_q_to_pred_str_q
saver.restore(sess, model_path)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 1292, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]
Caused by op 'save/RestoreV2', defined at:
File "/data/MAB/miniconda3/envs/deepsignalenv/bin/deepsignal", line 33, in <module>
sys.exit(load_entry_point('deepsignal==0.2.0', 'console_scripts', 'deepsignal')())
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 424, in main
args.func(args)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/deepsignal.py", line 88, in main_call_mods
f5_args)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 448, in call_mods
is_rnn, is_base, is_cnn)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 326, in _call_mods_from_fast5s_cpu
p.start()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py", line 254, in _fast5s_q_to_pred_str_q
saver = tf.train.Saver()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
self.build()
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 881, in _build
build_save=build_save, build_restore=build_restore)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
restore_sequentially, reshape)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
restore_sequentially)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
name=name)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
self._traceback = tf_stack.extract_stack()
NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key modelem/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[node save/RestoreV2 (defined at /data/MAB/miniconda3/envs/deepsignalenv/lib/python3.7/site-packages/deepsignal-0.2.0-py3.7.egg/deepsignal/call_modifications.py:254) ]]
write process 1352722 ending
0 of 110340 fast5 files failed..
call_mods costs 26.37 seconds..The results file (single_barcode12.gatc.call_mods.tsv) that is produced is empty.
I am downloading the trained models, but I am not getting any files ending with .ckpt.
Here are the contents of one of the directories:
Which of these files should i provide to the --model_path command?