tensorflow / tensorrt

TensorFlow/TensorRT integration
Apache License 2.0
737 stars 226 forks source link

Accuracy drops a lot,when after build TensorRTengine #268

Open zeqinLi opened 3 years ago

zeqinLi commented 3 years ago

I am trying to convert the keras‘ model to TensorRTengine. My model contains CNN layer and GRU layer. And I use tensorrt python API to do this.At the same time, I build the engine successfully.But accuracy drops a lot.And I can't find what caused it.

Through testing, I found that the output of the layer before the GRU layer is consistent with the output of keras.But after adding the GRU layer, the output is much different. So I can confirm that the problem is caused by the GRU layer.But I can’t find a specific reason.This problem has bothered me for a long time.

keras code show as below:

from keras.layers import Bidirectional, GRU
gru_layer = Bidirectional(GRU(256, return_sequences=True), name='gru_layer')(input)

tensorrt code show as below:

def add_bigru(self,input_layer,layer_name,layer_index,nhidden=1):
  gru_1 = self.network.add_rnn_v2(input_layer.get_output(0), layer_count=1, hidden_size=256,
  max_seq_length=input_layer.get_output(0).shape[1], op=trt.RNNOperation.GRU)
  gru_1.direction = trt.RNNDirection.BIDIRECTION
  # forward
  forward_kernel_weight_unit, forward_recurrent_kernel_weight_unit, forward_bias_weight_unit = self.split_gru_weight(
      [self.pb_weight[layer_name+"/forward_gru_"+str(layer_index)+"/kernel"], self.pb_weight[layer_name+"/forward_gru_"+str(layer_index)+"/recurrent_kernel"],
       self.pb_weight[layer_name+"/forward_gru_"+str(layer_index)+"/bias"]], 3)

  # backward
  backward_kernel_weight_unit, backward_recurrent_kernel_weight_unit, backward_bias_weight_unit = self.split_gru_weight(
      [self.pb_weight[layer_name+"/backward_gru_"+str(layer_index)+"/kernel"], self.pb_weight[layer_name+"/backward_gru_"+str(layer_index)+"/recurrent_kernel"],
       self.pb_weight[layer_name+"/backward_gru_"+str(layer_index)+"/bias"]], 3)

  for i in range(2):

      if i == 0:
          kernel_weight_unit, recurrent_kernel_weight_unit, bias_weight_unit = forward_kernel_weight_unit, forward_recurrent_kernel_weight_unit, forward_bias_weight_unit
      else:
          kernel_weight_unit, recurrent_kernel_weight_unit, bias_weight_unit = backward_kernel_weight_unit, backward_recurrent_kernel_weight_unit, backward_bias_weight_unit

      layer_index = i
      # update gate
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.UPDATE, is_w=True,
                                 weights=kernel_weight_unit[str(0)])
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.UPDATE, is_w=False,
                                 weights=recurrent_kernel_weight_unit[str(0)])
      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.UPDATE, is_w=True,
                              bias=bias_weight_unit[str(0)])
      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.UPDATE, is_w=False,
                              bias=np.zeros(256).astype(np.float32))

      # reset gate
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.RESET, is_w=True,
                                 weights=kernel_weight_unit[str(1)])
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.RESET, is_w=False,
                                 weights=recurrent_kernel_weight_unit[str(1)])

      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.RESET, is_w=True,
                              bias=bias_weight_unit[str(1)])
      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.RESET, is_w=False,
                              bias=np.zeros(256).astype(np.float32))

      # hidden gate
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.HIDDEN, is_w=True,
                                 weights=kernel_weight_unit[str(2)])
      gru_1.set_weights_for_gate(layer_index=layer_index, gate=trt.RNNGateType.HIDDEN, is_w=False,
                                 weights=recurrent_kernel_weight_unit[str(2)])

      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.HIDDEN, is_w=True,
                              bias=bias_weight_unit[str(2)])
      gru_1.set_bias_for_gate(layer_index=layer_index, gate=trt.RNNGateType.HIDDEN, is_w=False,
                              bias=np.zeros(256).astype(np.float32))
  return  gru_1

Environment:

TensorRT Version: 7.0.0.11
NVIDIA GPU: Tesla P100
NVIDIA Driver Version: 440.118.02
CUDA Version:
CUDNN Version: cudatoolkit 10.0.130
Operating System: centos
Python Version (if applicable): 3.6.13
Tensorflow Version (if applicable): 1,13,1
Keras Version (if applicable): 2.1.6

Can anyone help me solve it?

zeqinLi commented 3 years ago

refer to : https://github.com/NVIDIA/TensorRT/issues/1443