SlienceSpring
commented
3 years ago 检查一下cudnn？
Open lbq779660843 opened 3 years ago
SlienceSpring
commented
3 years ago 检查一下cudnn？
fire717
commented
3 years ago 我也遇到了这个问题，试了下，注释这一句就可以了 config.gpu_options.per_process_gpu_memory_fraction = 0.95
然后遇到了OOM报错，降低batchsize了只有，降到8跑起来了
E:\Anaconda3\envs\nlp\python.exe F:/99_projects/codes/ASRT_v0.6.1/SpeechModel251.py Using TensorFlow backend. E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) E:\Anaconda3\envs\nlp\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) WARNING:tensorflow:From E:\Anaconda3\envs\nlp\lib\site-packages\keras\backend\tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\ops\array_ops.py:1354: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Model: "model_2"
Layer (type) Output Shape Param # Connected to
the_input (InputLayer) (None, 1600, 200, 1) 0
conv2d_1 (Conv2D) (None, 1600, 200, 32 288 the_input[0][0]
dropout_1 (Dropout) (None, 1600, 200, 32 0 conv2d_1[0][0]
conv2d_2 (Conv2D) (None, 1600, 200, 32 9248 dropout_1[0][0]
max_pooling2d_1 (MaxPooling2D) (None, 800, 100, 32) 0 conv2d_2[0][0]
dropout_2 (Dropout) (None, 800, 100, 32) 0 max_pooling2d_1[0][0]
conv2d_3 (Conv2D) (None, 800, 100, 64) 18496 dropout_2[0][0]
dropout_3 (Dropout) (None, 800, 100, 64) 0 conv2d_3[0][0]
conv2d_4 (Conv2D) (None, 800, 100, 64) 36928 dropout_3[0][0]
max_pooling2d_2 (MaxPooling2D) (None, 400, 50, 64) 0 conv2d_4[0][0]
dropout_4 (Dropout) (None, 400, 50, 64) 0 max_pooling2d_2[0][0]
conv2d_5 (Conv2D) (None, 400, 50, 128) 73856 dropout_4[0][0]
dropout_5 (Dropout) (None, 400, 50, 128) 0 conv2d_5[0][0]
conv2d_6 (Conv2D) (None, 400, 50, 128) 147584 dropout_5[0][0]
max_pooling2d_3 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_6[0][0]
dropout_6 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_3[0][0]
conv2d_7 (Conv2D) (None, 200, 25, 128) 147584 dropout_6[0][0]
dropout_7 (Dropout) (None, 200, 25, 128) 0 conv2d_7[0][0]
conv2d_8 (Conv2D) (None, 200, 25, 128) 147584 dropout_7[0][0]
max_pooling2d_4 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_8[0][0]
dropout_8 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_4[0][0]
conv2d_9 (Conv2D) (None, 200, 25, 128) 147584 dropout_8[0][0]
dropout_9 (Dropout) (None, 200, 25, 128) 0 conv2d_9[0][0]
conv2d_10 (Conv2D) (None, 200, 25, 128) 147584 dropout_9[0][0]
max_pooling2d_5 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_10[0][0]
reshape_1 (Reshape) (None, 200, 3200) 0 max_pooling2d_5[0][0]
dropout_10 (Dropout) (None, 200, 3200) 0 reshape_1[0][0]
dense_1 (Dense) (None, 200, 128) 409728 dropout_10[0][0]
dropout_11 (Dropout) (None, 200, 128) 0 dense_1[0][0]
dense_2 (Dense) (None, 200, 1424) 183696 dropout_11[0][0]
Activation0 (Activation) (None, 200, 1424) 0 dense_2[0][0]
the_labels (InputLayer) (None, 64) 0
input_length (InputLayer) (None, 1) 0
label_length (InputLayer) (None, 1) 0
ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]
Total params: 1,470,160 Trainable params: 1,470,160 Non-trainable params: 0
2020-12-23 09:12:13.529531: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2020-12-23 09:12:13.531659: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll 2020-12-23 09:12:13.554182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2070 major: 7 minor: 5 memoryClockRate(GHz): 1.71 pciBusID: 0000:01:00.0 2020-12-23 09:12:13.554307: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check. 2020-12-23 09:12:13.554720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2020-12-23 09:12:14.056482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-23 09:12:14.056576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2020-12-23 09:12:14.056625: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2020-12-23 09:12:14.057082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6314 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070, pci bus id: 0000:01:00.0, compute capability: 7.5) WARNING:tensorflow:From E:\Anaconda3\envs\nlp\lib\site-packages\keras\backend\tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[*Info] Create Model Successful, Compiles Model Successful. yes 2020-12-23 09:12:15.960754: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED 2020-12-23 09:12:15.960844: E tensorflow/stream_executor/cuda/cuda_dnn.cc:333] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows 2020-12-23 09:12:15.961471: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED 2020-12-23 09:12:15.961548: E tensorflow/stream_executor/cuda/cuda_dnn.cc:333] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows Traceback (most recent call last): File "F:/99_projects/codes/ASRT_v0.6.1/SpeechModel251.py", line 441, in
r = ms.RecognizeSpeech_FromFile('D:\NLP\ST-CMDS-20170001_1-OS\20170001P00241I0053.wav')
File "F:/99_projects/codes/ASRT_v0.6.1/SpeechModel251.py", line 382, in RecognizeSpeech_FromFile
r = self.RecognizeSpeech(wavsignal, fs)
File "F:/99_projects/codes/ASRT_v0.6.1/SpeechModel251.py", line 362, in RecognizeSpeech
r1 = self.Predict(data_input, input_length)
File "F:/99_projects/codes/ASRT_v0.6.1/SpeechModel251.py", line 304, in Predict
base_pred = self.base_model.predict(x = x_in)
File "E:\Anaconda3\envs\nlp\lib\site-packages\keras\engine\training.py", line 1462, in predict
callbacks=callbacks)
File "E:\Anaconda3\envs\nlp\lib\site-packages\keras\engine\training_arrays.py", line 324, in predict_loop
batch_outs = f(ins_batch)
File "E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\keras\backend.py", line 3292, in call
run_metadata=self.run_metadata)
File "E:\Anaconda3\envs\nlp\lib\site-packages\tensorflow\python\client\session.py", line 1458, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv2d_1/convolution}}]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv2d_1/convolution}}]]
[[Activation0/truediv/_63]]
0 successful operations.
0 derived errors ignored.
Process finished with exit code 1