myrainbowandsky commented 5 years ago

系统：2块GTX1080Ti，UBUNTU16.04，PYTHON3.5，TENSORFLOW1.12.0 报错如下，不管哪个多GPU模型一样的错误。 Using TensorFlow backend. 2019-03-27 17:52:14.445258: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2019-03-27 17:52:14.521421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645 pciBusID: 0000:17:00.0 totalMemory: 10.92GiB freeMemory: 10.77GiB 2019-03-27 17:52:14.521448: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2019-03-27 17:52:14.847342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-03-27 17:52:14.847373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-03-27 17:52:14.847380: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-03-27 17:52:14.847688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10619 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)

Layer (type) Output Shape Param # Connected to

the_input (InputLayer) (None, 1600, 200, 1) 0

conv2d_1 (Conv2D) (None, 1600, 200, 32 288 the_input[0][0]

conv2d_2 (Conv2D) (None, 1600, 200, 32 9248 conv2d_1[0][0]

max_pooling2d_1 (MaxPooling2D) (None, 800, 100, 32) 0 conv2d_2[0][0]

conv2d_3 (Conv2D) (None, 800, 100, 64) 18496 max_pooling2d_1[0][0]

conv2d_4 (Conv2D) (None, 800, 100, 64) 36928 conv2d_3[0][0]

max_pooling2d_2 (MaxPooling2D) (None, 400, 50, 64) 0 conv2d_4[0][0]

conv2d_5 (Conv2D) (None, 400, 50, 128) 73856 max_pooling2d_2[0][0]

conv2d_6 (Conv2D) (None, 400, 50, 128) 147584 conv2d_5[0][0]

max_pooling2d_3 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_6[0][0]

conv2d_7 (Conv2D) (None, 200, 25, 128) 147584 max_pooling2d_3[0][0]

conv2d_8 (Conv2D) (None, 200, 25, 128) 147584 conv2d_7[0][0]

max_pooling2d_4 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_8[0][0]

conv2d_9 (Conv2D) (None, 200, 25, 128) 147584 max_pooling2d_4[0][0]

conv2d_10 (Conv2D) (None, 200, 25, 128) 147584 conv2d_9[0][0]

max_pooling2d_5 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_10[0][0]

reshape_1 (Reshape) (None, 200, 3200) 0 max_pooling2d_5[0][0]

dense_1 (Dense) (None, 200, 128) 409728 reshape_1[0][0]

gru1 (GRU) (None, 200, 128) 98688 dense_1[0][0]

gru1_b (GRU) (None, 200, 128) 98688 dense_1[0][0]

add_1 (Add) (None, 200, 128) 0 gru1[0][0]
gru1_b[0][0]

gru2 (GRU) (None, 200, 128) 98688 add_1[0][0]

gru2_b (GRU) (None, 200, 128) 98688 add_1[0][0]

concatenate_1 (Concatenate) (None, 200, 256) 0 gru2[0][0]
gru2_b[0][0]

dense_2 (Dense) (None, 200, 128) 32896 concatenate_1[0][0]

dense_3 (Dense) (None, 200, 1422) 183438 dense_2[0][0]

Activation0 (Activation) (None, 200, 1422) 0 dense_3[0][0]

the_labels (InputLayer) (None, 64) 0

input_length (InputLayer) (None, 1) 0

label_length (InputLayer) (None, 1) 0

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

Total params: 1,897,550 Trainable params: 1,897,550 Non-trainable params: 0

[Info] Create Model Successful, Compiles Model Successful. [running] train epoch 0 . [message] epoch 0 . Have train datas 0+ Epoch 1/1 Traceback (most recent call last): File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(args) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _run_fn self._extend_graph() File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1352, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation tower_0/model_2/ctc/ToInt32: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU XLA_CPU XLA_GPU TensorArrayReadV3: CPU XLA_CPU XLA_GPU Enter: GPU CPU XLA_CPU XLA_GPU TensorArrayV3: CPU XLA_CPU XLA_GPU Cast: GPU CPU XLA_CPU XLA_GPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='XLA_CPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_CPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='CPU' device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL]

 [[{{node tower_0/model_2/ctc/ToInt32}} = Cast[DstT=DT_INT32, SrcT=DT_INT64, Truncate=false, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_mspeech.py", line 53, in ms.TrainModel(datapath, epoch = 500, batch_size = 20, save_step = 500) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel261_p.py", line 200, in TrainModel self._model.fit_generator(yielddatas, save_step) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1418, in fit_generator initial_epoch=initial_epoch) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/training_generator.py", line 217, in fit_generator class_weight=class_weight) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1217, in train_on_batch outputs = self.train_function(ins) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2697, in call if hasattr(get_session(), '_make_callable_from_options'): File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 199, in get_session [tf.is_variable_initialized(v) for v in candidate_vars]) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation tower_0/model_2/ctc/ToInt32: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU XLA_CPU XLA_GPU TensorArrayReadV3: CPU XLA_CPU XLA_GPU Enter: GPU CPU XLA_CPU XLA_GPU TensorArrayV3: CPU XLA_CPU XLA_GPU Cast: GPU CPU XLA_CPU XLA_GPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='XLA_CPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_CPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='CPU' device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL]

 [[node tower_0/model_2/ctc/ToInt32 (defined at /home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:4249)  = Cast[DstT=DT_INT32, SrcT=DT_INT64, Truncate=false, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

Caused by op 'tower_0/model_2/ctc/ToInt32', defined at: File "train_mspeech.py", line 50, in ms = ModelSpeech(datapath) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel261_p.py", line 44, in init self._model, self.base_model = self.CreateModel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel261_p.py", line 155, in CreateModel model = ParallelModel(model, NUM_GPU) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 30, in init merged_outputs = self.make_parallel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 74, in make_parallel outputs = self.inner_model(inputs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/base_layer.py", line 457, in call output = self.call(inputs, kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 564, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 739, in run_internal_graph layer.call(computed_tensors, kwargs)) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/layers/core.py", line 687, in call return self.function(inputs, *arguments) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel261_p.py", line 172, in ctc_lambda_func return K.ctc_batch_cost(labels, y_pred, input_length, label_length) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 4249, in ctc_batch_cost label_length = tf.to_int32(tf.squeeze(label_length, axis=-1)) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 768, in to_int32 return cast(x, dtypes.int32, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 677, in cast x = gen_math_ops.cast(x, base_type, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1659, in cast "Cast", x=x, DstT=DstT, Truncate=Truncate, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func return func(args, **kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1770, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Cannot assign a device for operation tower_0/model_2/ctc/ToInt32: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU XLA_CPU XLA_GPU TensorArrayReadV3: CPU XLA_CPU XLA_GPU Enter: GPU CPU XLA_CPU XLA_GPU TensorArrayV3: CPU XLA_CPU XLA_GPU Cast: GPU CPU XLA_CPU XLA_GPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='XLA_CPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_CPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BOOL, DT_QINT8, DT_QUINT8, DT_QINT32, DT_HALF, DT_UINT32, DT_UINT64] device='XLA_GPU_JIT'; DstT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64]; SrcT in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT8, ..., DT_QINT32, DT_BFLOAT16, DT_HALF, DT_UINT32, DT_UINT64] device='CPU' device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL]

 [[node tower_0/model_2/ctc/ToInt32 (defined at /home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py:4249)  = Cast[DstT=DT_INT32, SrcT=DT_INT64, Truncate=false, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

nl8590687 commented 5 years ago

我这里是没有任何问题的，而且我也从来没有遇到过这种问题，请首先检查一下tensorflow和keras以及Cuda和cudnn的配置，也许是配置有误。另外，看到你使用的tensorflow的版本是1.12.0，我觉得也有可能是新版tensorflow的api的变化导致的，请降级到1.11.0或者更早的版本试试。

myrainbowandsky commented 5 years ago

我这里是没有任何问题的，而且我也从来没有遇到过这种问题，请首先检查一下tensorflow和keras以及Cuda和cudnn的配置，也许是配置有误。另外，看到你使用的tensorflow的版本是1.12.0，我觉得也有可能是新版tensorflow的api的变化导致的，请降级到1.11.0或者更早的版本试试。

我试试看，再来回复

myrainbowandsky commented 5 years ago

降级到1.10.0 还是报错：

Using TensorFlow backend. 2019-03-29 17:29:42.156915: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2019-03-29 17:29:42.244375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645 pciBusID: 0000:17:00.0 totalMemory: 10.92GiB freeMemory: 10.77GiB 2019-03-29 17:29:42.244402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0 2019-03-29 17:29:42.561789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-03-29 17:29:42.561817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0 2019-03-29 17:29:42.561822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N 2019-03-29 17:29:42.563806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10619 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)

Layer (type) Output Shape Param # Connected to

the_input (InputLayer) (None, 1600, 200, 1) 0

conv2d_1 (Conv2D) (None, 1600, 200, 32 288 the_input[0][0]

dropout_1 (Dropout) (None, 1600, 200, 32 0 conv2d_1[0][0]

conv2d_2 (Conv2D) (None, 1600, 200, 32 9248 dropout_1[0][0]

max_pooling2d_1 (MaxPooling2D) (None, 800, 100, 32) 0 conv2d_2[0][0]

dropout_2 (Dropout) (None, 800, 100, 32) 0 max_pooling2d_1[0][0]

conv2d_3 (Conv2D) (None, 800, 100, 64) 18496 dropout_2[0][0]

dropout_3 (Dropout) (None, 800, 100, 64) 0 conv2d_3[0][0]

conv2d_4 (Conv2D) (None, 800, 100, 64) 36928 dropout_3[0][0]

max_pooling2d_2 (MaxPooling2D) (None, 400, 50, 64) 0 conv2d_4[0][0]

dropout_4 (Dropout) (None, 400, 50, 64) 0 max_pooling2d_2[0][0]

conv2d_5 (Conv2D) (None, 400, 50, 128) 73856 dropout_4[0][0]

dropout_5 (Dropout) (None, 400, 50, 128) 0 conv2d_5[0][0]

conv2d_6 (Conv2D) (None, 400, 50, 128) 147584 dropout_5[0][0]

max_pooling2d_3 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_6[0][0]

dropout_6 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_3[0][0]

conv2d_7 (Conv2D) (None, 200, 25, 128) 147584 dropout_6[0][0]

dropout_7 (Dropout) (None, 200, 25, 128) 0 conv2d_7[0][0]

conv2d_8 (Conv2D) (None, 200, 25, 128) 147584 dropout_7[0][0]

max_pooling2d_4 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_8[0][0]

dropout_8 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_4[0][0]

conv2d_9 (Conv2D) (None, 200, 25, 128) 147584 dropout_8[0][0]

dropout_9 (Dropout) (None, 200, 25, 128) 0 conv2d_9[0][0]

conv2d_10 (Conv2D) (None, 200, 25, 128) 147584 dropout_9[0][0]

max_pooling2d_5 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_10[0][0]

reshape_1 (Reshape) (None, 200, 3200) 0 max_pooling2d_5[0][0]

dropout_10 (Dropout) (None, 200, 3200) 0 reshape_1[0][0]

dense_1 (Dense) (None, 200, 128) 409728 dropout_10[0][0]

dropout_11 (Dropout) (None, 200, 128) 0 dense_1[0][0]

dense_2 (Dense) (None, 200, 1424) 183696 dropout_11[0][0]

Activation0 (Activation) (None, 200, 1424) 0 dense_2[0][0]

the_labels (InputLayer) (None, 64) 0

input_length (InputLayer) (None, 1) 0

label_length (InputLayer) (None, 1) 0

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

Total params: 1,470,160 Trainable params: 1,470,160 Non-trainable params: 0

[Info] Create Model Successful, Compiles Model Successful. Traceback (most recent call last): File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call return fn(args) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1261, in _run_fn self._extend_graph() File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1295, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_mspeech.py", line 46, in ms.LoadModel(modelpath + '/m251/speech_model251_e_0_step_163000.model') File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 197, in LoadModel self._model.load_weights(filename) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 1166, in load_weights f, self.layers, reshape=reshape) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group K.batch_set_value(weight_value_tuples) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value get_session().run(assign_ops, feed_dict=feed_dict) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 199, in get_session [tf.is_variable_initialized(v) for v in candidate_vars]) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 877, in run run_metadata_ptr) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1100, in _run feed_dict_tensor, options, run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

Caused by op 'tower_0/model_2/ctc/ToInt32', defined at: File "train_mspeech.py", line 44, in ms = ModelSpeech(datapath) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 43, in init self._model, self.base_model = self.CreateModel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 138, in CreateModel model = ParallelModel(model, NUM_GPU) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 30, in init merged_outputs = self.make_parallel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 74, in make_parallel outputs = self.inner_model(inputs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/base_layer.py", line 457, in call output = self.call(inputs, kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 564, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 739, in run_internal_graph layer.call(computed_tensors, kwargs)) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/layers/core.py", line 687, in call return self.function(inputs, *arguments) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 155, in ctc_lambda_func return K.ctc_batch_cost(labels, y_pred, input_length, label_length) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 4249, in ctc_batch_cost label_length = tf.to_int32(tf.squeeze(label_length, axis=-1)) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 756, in to_int32 return cast(x, dtypes.int32, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 668, in cast x = gen_math_ops.cast(x, base_type, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1645, in cast "Cast", x=x, DstT=DstT, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func return func(args, **kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

我这里是没有任何问题的，而且我也从来没有遇到过这种问题，请首先检查一下tensorflow和keras以及Cuda和cudnn的配置，也许是配置有误。另外，看到你使用的tensorflow的版本是1.12.0，我觉得也有可能是新版tensorflow的api的变化导致的，请降级到1.11.0或者更早的版本试试。

nl8590687 commented 5 years ago

那就试试降到tensorflow 1.4.0版？我这里用1.4.0还有1.9.0通过GPU运行都没问题

myrainbowandsky commented 5 years ago

降级到1.09还是报错：

Using TensorFlow backend. 2019-04-01 16:33:37.484921: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2019-04-01 16:33:37.576872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.645 pciBusID: 0000:17:00.0 totalMemory: 10.92GiB freeMemory: 10.77GiB 2019-04-01 16:33:37.576899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0 2019-04-01 16:33:37.876290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-04-01 16:33:37.876321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0 2019-04-01 16:33:37.876327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N 2019-04-01 16:33:37.876620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10619 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:17:00.0, compute capability: 6.1)

Layer (type) Output Shape Param # Connected to

the_input (InputLayer) (None, 1600, 200, 1) 0

conv2d_1 (Conv2D) (None, 1600, 200, 32 288 the_input[0][0]

dropout_1 (Dropout) (None, 1600, 200, 32 0 conv2d_1[0][0]

conv2d_2 (Conv2D) (None, 1600, 200, 32 9248 dropout_1[0][0]

max_pooling2d_1 (MaxPooling2D) (None, 800, 100, 32) 0 conv2d_2[0][0]

dropout_2 (Dropout) (None, 800, 100, 32) 0 max_pooling2d_1[0][0]

conv2d_3 (Conv2D) (None, 800, 100, 64) 18496 dropout_2[0][0]

dropout_3 (Dropout) (None, 800, 100, 64) 0 conv2d_3[0][0]

conv2d_4 (Conv2D) (None, 800, 100, 64) 36928 dropout_3[0][0]

max_pooling2d_2 (MaxPooling2D) (None, 400, 50, 64) 0 conv2d_4[0][0]

dropout_4 (Dropout) (None, 400, 50, 64) 0 max_pooling2d_2[0][0]

conv2d_5 (Conv2D) (None, 400, 50, 128) 73856 dropout_4[0][0]

dropout_5 (Dropout) (None, 400, 50, 128) 0 conv2d_5[0][0]

conv2d_6 (Conv2D) (None, 400, 50, 128) 147584 dropout_5[0][0]

max_pooling2d_3 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_6[0][0]

dropout_6 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_3[0][0]

conv2d_7 (Conv2D) (None, 200, 25, 128) 147584 dropout_6[0][0]

dropout_7 (Dropout) (None, 200, 25, 128) 0 conv2d_7[0][0]

conv2d_8 (Conv2D) (None, 200, 25, 128) 147584 dropout_7[0][0]

max_pooling2d_4 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_8[0][0]

dropout_8 (Dropout) (None, 200, 25, 128) 0 max_pooling2d_4[0][0]

conv2d_9 (Conv2D) (None, 200, 25, 128) 147584 dropout_8[0][0]

dropout_9 (Dropout) (None, 200, 25, 128) 0 conv2d_9[0][0]

conv2d_10 (Conv2D) (None, 200, 25, 128) 147584 dropout_9[0][0]

max_pooling2d_5 (MaxPooling2D) (None, 200, 25, 128) 0 conv2d_10[0][0]

reshape_1 (Reshape) (None, 200, 3200) 0 max_pooling2d_5[0][0]

dropout_10 (Dropout) (None, 200, 3200) 0 reshape_1[0][0]

dense_1 (Dense) (None, 200, 128) 409728 dropout_10[0][0]

dropout_11 (Dropout) (None, 200, 128) 0 dense_1[0][0]

dense_2 (Dense) (None, 200, 1424) 183696 dropout_11[0][0]

Activation0 (Activation) (None, 200, 1424) 0 dense_2[0][0]

the_labels (InputLayer) (None, 64) 0

input_length (InputLayer) (None, 1) 0

label_length (InputLayer) (None, 1) 0

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

Total params: 1,470,160 Trainable params: 1,470,160 Non-trainable params: 0

[Info] Create Model Successful, Compiles Model Successful. Traceback (most recent call last): File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call return fn(args) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1305, in _run_fn self._extend_graph() File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _extend_graph tf_session.ExtendSession(self._session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_mspeech.py", line 46, in ms.LoadModel(modelpath + '/m251/speech_model251_e_0_step_228500.model') File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 197, in LoadModel self._model.load_weights(filename) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 1166, in load_weights f, self.layers, reshape=reshape) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group K.batch_set_value(weight_value_tuples) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value get_session().run(assign_ops, feed_dict=feed_dict) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 199, in get_session [tf.is_variable_initialized(v) for v in candidate_vars]) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

Caused by op 'tower_0/model_2/ctc/ToInt32', defined at: File "train_mspeech.py", line 44, in ms = ModelSpeech(datapath) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 43, in init self._model, self.base_model = self.CreateModel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 138, in CreateModel model = ParallelModel(model, NUM_GPU) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 30, in init merged_outputs = self.make_parallel() File "/home/sky-ai/xwt/ASRT_SpeechRecognition/general_function/muti_gpu.py", line 74, in make_parallel outputs = self.inner_model(inputs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/base_layer.py", line 457, in call output = self.call(inputs, kwargs) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 564, in call outputtensors, , _ = self.run_internal_graph(inputs, masks) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/engine/network.py", line 739, in run_internal_graph layer.call(computed_tensors, kwargs)) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/layers/core.py", line 687, in call return self.function(inputs, **arguments) File "/home/sky-ai/xwt/ASRT_SpeechRecognition/SpeechModel251_p.py", line 155, in ctc_lambda_func return K.ctc_batch_cost(labels, y_pred, input_length, label_length) File "/home/sky-ai/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 4249, in ctc_batch_cost label_length = tf.to_int32(tf.squeeze(label_length, axis=-1)) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 753, in to_int32 return cast(x, dtypes.int32, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 665, in cast x = gen_math_ops.cast(x, base_type, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1525, in cast "Cast", x=x, DstT=DstT, name=name) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op op_def=op_def) File "/home/sky-ai/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1740, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'tower_0/model_2/ctc/ToInt32': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Colocation Debug Info: Colocation group had the following types and devices: TensorArrayScatterV3: CPU TensorArrayReadV3: CPU Enter: GPU CPU TensorArrayV3: CPU Cast: GPU CPU

Colocation members and user-requested devices: tower_0/model_2/ctc/ToInt32 (Cast) /device:GPU:0 tower_0/model_2/ctc/scan/TensorArray (TensorArrayV3) tower_0/model_2/ctc/scan/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 (TensorArrayScatterV3) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3/Enter (Enter) /device:GPU:0 tower_0/model_2/ctc/scan/while/TensorArrayReadV3 (TensorArrayReadV3) /device:GPU:0

Registered kernels: device='GPU'; SrcT in [DT_BFLOAT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BFLOAT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX128]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_COMPLEX64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_DOUBLE]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_FLOAT]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_HALF]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT64]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT32]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT16]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_INT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_UINT8]; DstT in [DT_BOOL] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX128] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_COMPLEX64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_DOUBLE] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_FLOAT] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_HALF] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT64] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT32] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT16] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_INT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_UINT8] device='GPU'; SrcT in [DT_BOOL]; DstT in [DT_BOOL] device='CPU'

 [[Node: tower_0/model_2/ctc/ToInt32 = Cast[DstT=DT_INT32, SrcT=DT_INT64, _device="/device:GPU:0"](tower_0/model_2/ctc/Squeeze)]]

myrainbowandsky commented 5 years ago

补坑：SpeechModel251_p.py 这样设置通过( tf1.09) if(name=='main'):

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
#os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
#进行配置，使用70%的GPU
config = tf.ConfigProto()
#config.gpu_options.per_process_gpu_memory_fraction = 0.95
#config.gpu_options.allow_growth=True   #不全部占满显存, 按需分配
#set_session(tf.Session(config=config))

xuchenxing commented 5 years ago

我也遇到同样的问题了，能否把你的解决方法详细说下？我照着你这个改还是不行。谢谢！@myrainbowandsky

补坑：SpeechModel251_p.py 这样设置通过( tf1.09) if(name=='main'):

import tensorflow as tf
from keras.backend.tensorflow_backend import set_session
#os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
#进行配置，使用70%的GPU
config = tf.ConfigProto()
#config.gpu_options.per_process_gpu_memory_fraction = 0.95
#config.gpu_options.allow_growth=True   #不全部占满显存, 按需分配
#set_session(tf.Session(config=config))

haixiaotian commented 5 years ago

我遇到同样问题，请问解决了吗

nl8590687 / ASRT_SpeechRecognition

多GPU问题报错，2块GTX1080TI #92

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

nl8590687 / ASRT_SpeechRecognition

多GPU问题报错，2块GTX1080TI #92

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0] the_labels[0][0] input_length[0][0] label_length[0][0]

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0] the_labels[0][0] input_length[0][0] label_length[0][0]

Layer (type) Output Shape Param # Connected to

ctc (Lambda) (None, 1) 0 Activation0[0][0] the_labels[0][0] input_length[0][0] label_length[0][0]

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]

ctc (Lambda) (None, 1) 0 Activation0[0][0]
the_labels[0][0]
input_length[0][0]
label_length[0][0]