search

google-coral / edgetpu

Coral issue tracker (and legacy Edge TPU API source)
https://coral.ai
Apache License 2.0
428 stars 125 forks source link

EfficientDet Lite 3 WITH TRANSFER LEARNING: Compilation failed due to large activation tensors in model #653

Closed ghbn8 closed 2 years ago

ghbn8 commented 2 years ago

Description

I have a EfficientDet Lite 3 model that I have retrained with TRANSFER LEARNING (code below), that I am attempting to compile using the Google Colab notebook provided by Google https://colab.research.google.com/github/google-coral/tutorials/blob/master/compile_for_edgetpu.ipynb.

The colab notebook provided above failed due to large activation tensors in model., I then ran the following compilation command with the included flags edgetpu_compiler -s -d -a model_efficientdet_lite3_detection_metadata_1.tflite -- this command ran for several hours, attempting to compile and removing operations at every step. The model never compiled, each time providing an error similar to below:

Try to compile segment with 421 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,Reshape_9 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed!

My questions are:

  1. Why is this model failing to compile?
  2. I am able to compile the EfficientDet Lite 3 model WITHOUT TRANSFER LEARNING (using the flags above); applying transfer learning should not change the structure of the neural network considerably, should it?
  3. Can you provide a recommendation for how to compile this model

The code snippet I used for transfer learning and exporting the model is below:

BATCH_SIZE = 5; EPOCHS = 1; MAX_NUM_IMAGES = 50
train_data = object_detector.DataLoader.from_pascal_voc('train', 'train', label_map=['obstacle', 'null'], max_num_images=MAX_NUM_IMAGES)
validate_data = object_detector.DataLoader.from_pascal_voc('validate', 'validate', label_map=['obstacle', 'null'], max_num_images=MAX_NUM_IMAGES)
test_data = object_detector.DataLoader.from_pascal_voc('test', 'test', label_map=['obstacle', 'null'], max_num_images=10)

spec = model_spec.get('efficientdet_lite3') #Only lite 2 and below will compile for the Coral Edge TPU

model = object_detector.create(train_data, model_spec=spec, batch_size=BATCH_SIZE, do_train=True, train_whole_model=False, epochs=EPOCHS, validation_data=validate_data)

model.export(export_dir='.', tflite_filename=TFLITE_MODEL_NAME)

NOTE: EfficientDet Lite models 0, 1, 2 WITH and WITHOUT TRANSFER LEARNING compiled without an error.

Thank you

Click to expand! ### Issue Type _No response_ ### Operating System _No response_ ### Coral Device _No response_ ### Other Devices _No response_ ### Programming Language _No response_ ### Relevant Log Output ```shell Edge TPU Compiler version 16.0.384591198 Searching for valid delegate with step 1 Try to compile segment with 424 ops Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 423 ops Intermediate tensors: Sigmoid1,tfl.dequantize Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 422 ops Intermediate tensors: Sigmoid1,concat_1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 421 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,Reshape_9 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 420 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box_net/box-predict/BiasAdd_4;box_net/box-predict/separable_conv2d_4;box_net/box-predict/BiasAdd/ReadVariableOp11 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 419 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box_net/box-predict/separable_conv2d_4/depthwise1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 418 ops Intermediate tensors: Reshape_1,Relu6_19;Relu6_19;box-3-bn-7/FusedBatchNormV3;box-3-bn-7/FusedBatchNormV3;box-3/BiasAdd/ReadVariableOp;box-3/BiasAdd/ReadVariableOp;box-3/BiasAdd_4;box-3/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;box-3/separable_conv2d_4;box-3/separable_conv2d_41,Reshape_7,Sigmoid1,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 417 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box-3/separable_conv2d_4/depthwise;box-3/separable_conv2d_4/depthwise1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 416 ops Intermediate tensors: Relu6_18;Relu6_18;box-2-bn-7/FusedBatchNormV3;box-2-bn-7/FusedBatchNormV3;box-2/BiasAdd/ReadVariableOp;box-2/BiasAdd/ReadVariableOp;box-2/BiasAdd_4;box-2/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;box-2/separable_conv2d_4;box-2/separable_conv2d_41,Sigmoid1,Reshape_7,Reshape_5,Reshape_3,Reshape_1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 415 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box-2/separable_conv2d_4/depthwise;box-2/separable_conv2d_4/depthwise1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 414 ops Intermediate tensors: Reshape_1,Relu6_17;Relu6_17;box-1-bn-7/FusedBatchNormV3;box-1-bn-7/FusedBatchNormV3;box-1/BiasAdd/ReadVariableOp;box-1/BiasAdd/ReadVariableOp;box-1/BiasAdd_4;box-1/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;box-1/separable_conv2d_4;box-1/separable_conv2d_41,Reshape_7,Sigmoid1,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 413 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box-1/separable_conv2d_4/depthwise;box-1/separable_conv2d_4/depthwise1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 412 ops Intermediate tensors: Reshape_1,Relu6_16;Relu6_16;box-0-bn-7/FusedBatchNormV3;box-0-bn-7/FusedBatchNormV3;box-0/BiasAdd/ReadVariableOp;box-0/BiasAdd/ReadVariableOp;box-0/BiasAdd_4;box-0/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;box-0/separable_conv2d_4;box-0/separable_conv2d_41,Reshape_7,Sigmoid1,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 411 ops Intermediate tensors: Reshape_1,Reshape_7,Sigmoid1,Reshape_3,Reshape_5,box-0/separable_conv2d_4/depthwise;box-0/separable_conv2d_4/depthwise1 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 410 ops Intermediate tensors: Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_7,Sigmoid1,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 409 ops Intermediate tensors: Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_7,Sigmoid,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 408 ops Intermediate tensors: Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_7,concat,Reshape_3,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 407 ops Intermediate tensors: Reshape_5,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_6,Reshape_2,Reshape_7,Reshape_3,Reshape_8,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 406 ops Intermediate tensors: Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_5,Reshape_7,Reshape_2,Reshape_6,class_net/class-predict/BiasAdd_4;class_net/class-predict/separable_conv2d_4;class_net/class-predict/BiasAdd/ReadVariableOp11,Reshape_3,Reshape,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 405 ops Intermediate tensors: Reshape_5,Reshape,Reshape_4,Reshape_3,Reshape_7,class_net/class-predict/separable_conv2d_4/depthwise210,Reshape_2,Reshape_6,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 404 ops Intermediate tensors: Reshape,Reshape_5,Reshape_1,Reshape_2,Reshape_6,Reshape_7,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_3,Relu6_19;Relu6_19;class-3-bn-7/FusedBatchNormV3;class-3-bn-7/FusedBatchNormV3;class-3/BiasAdd/ReadVariableOp;class-3/BiasAdd/ReadVariableOp;class-3/BiasAdd_4;class-3/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;class-3/separable_conv2d_4;class-3/separable_conv2d_41,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 403 ops Intermediate tensors: Reshape,class-3/separable_conv2d_4/depthwise;class-3/separable_conv2d_4/depthwise1,Reshape_4,Reshape_3,Reshape_7,Reshape_6,Reshape_2,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_5 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 402 ops Intermediate tensors: Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_5,Reshape_7,Reshape_2,Reshape_6,Relu6_18;Relu6_18;class-2-bn-7/FusedBatchNormV3;class-2-bn-7/FusedBatchNormV3;class-2/BiasAdd/ReadVariableOp;class-2/BiasAdd/ReadVariableOp;class-2/BiasAdd_4;class-2/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;class-2/separable_conv2d_4;class-2/separable_conv2d_41,Reshape_3,Reshape,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 401 ops Intermediate tensors: Reshape_5,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_6,Reshape_2,Reshape_7,class-2/separable_conv2d_4/depthwise;class-2/separable_conv2d_4/depthwise1,Reshape_3,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 400 ops Intermediate tensors: Reshape_5,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_6,Reshape_2,Relu6_17;Relu6_17;class-1-bn-7/FusedBatchNormV3;class-1-bn-7/FusedBatchNormV3;class-1/BiasAdd/ReadVariableOp;class-1/BiasAdd/ReadVariableOp;class-1/BiasAdd_4;class-1/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;class-1/separable_conv2d_4;class-1/separable_conv2d_41,Reshape_7,Reshape_3,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 399 ops Intermediate tensors: Reshape_5,class-1/separable_conv2d_4/depthwise;class-1/separable_conv2d_4/depthwise1,Reshape_1,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_6,Reshape_2,Reshape_7,Reshape_3,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 398 ops Intermediate tensors: Reshape_1,Reshape_4,Reshape_5,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Relu6_16;Relu6_16;class-0-bn-7/FusedBatchNormV3;class-0-bn-7/FusedBatchNormV3;class-0/BiasAdd/ReadVariableOp;class-0/BiasAdd/ReadVariableOp;class-0/BiasAdd_4;class-0/BiasAdd_4;class_net/class-predict/separable_conv2d_4/depthwise;class-0/separable_conv2d_4;class-0/separable_conv2d_41,Reshape_7,Reshape_6,Reshape_2,Reshape_3,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 397 ops Intermediate tensors: Reshape,Reshape_5,Reshape_1,Reshape_2,Reshape_6,Reshape_7,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_3,class-0/separable_conv2d_4/depthwise;class-0/separable_conv2d_4/depthwise1,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 396 ops Intermediate tensors: Reshape_2,Reshape_6,Reshape_7,Reshape_5,Reshape_1,Reshape_4,cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_5/fnode7/op_after_combine12/conv/BiasAdd;cell_5/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d;cell_5/fnode7/op_after_combine12/conv/separable_conv2d,Reshape,Reshape_3 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 395 ops Intermediate tensors: Reshape_3,Reshape_7,Reshape_6,Reshape_2,Reshape_1,Reshape_5,Reshape,cell_5/fnode7/op_after_combine12/conv/separable_conv2d/depthwise;cell_5/fnode7/op_after_combine12/conv/separable_conv2d/depthwise1,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 394 ops Intermediate tensors: Reshape_5,Reshape_1,Reshape_6,Reshape_2,Reshape_7,Reshape_3,cell_5/fnode7/op_after_combine12/Relu6;cell_5/fnode7/op_after_combine12/Relu6;cell_5/fnode7/add_n/add;cell_5/fnode7/add_n/add,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 393 ops Intermediate tensors: Reshape,Reshape_5,Reshape_1,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_6,Reshape_7,Reshape_3,cell_5/fnode7/resample_1_11_12/max_pooling2d_23/MaxPool;cell_5/fnode7/resample_1_11_12/max_pooling2d_23/MaxPool,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 392 ops Intermediate tensors: Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_7,Reshape_5,Reshape_1,Reshape_4,Reshape,Reshape_3 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 391 ops Intermediate tensors: Reshape_1,Reshape_5,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_3,box_net/box-predict/BiasAdd_3;box_net/box-predict/separable_conv2d_4;box_net/box-predict/separable_conv2d_3;box_net/box-predict/BiasAdd/ReadVariableOp,Reshape,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 390 ops Intermediate tensors: Reshape_5,Reshape_1,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_2,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_3,box_net/box-predict/separable_conv2d_3/depthwise,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 389 ops Intermediate tensors: cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_2,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_5,Relu6_15;Relu6_15;box-3-bn-6/FusedBatchNormV3;box-3-bn-6/FusedBatchNormV3;box-3/BiasAdd/ReadVariableOp;box-3/BiasAdd/ReadVariableOp;box-3/BiasAdd_3;box-3/BiasAdd_3;class_net/class-predict/separable_conv2d_4/depthwise;box-3/separable_conv2d_3;box-3/separable_conv2d_31,Reshape_1,Reshape_4,Reshape,Reshape_3 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 388 ops Intermediate tensors: Reshape,Reshape_5,Reshape_1,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_3,box-3/separable_conv2d_3/depthwise;box-3/separable_conv2d_3/depthwise,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 387 ops Intermediate tensors: Reshape,Reshape_5,Relu6_14;Relu6_14;box-2-bn-6/FusedBatchNormV3;box-2-bn-6/FusedBatchNormV3;box-2/BiasAdd/ReadVariableOp;box-2/BiasAdd/ReadVariableOp;box-2/BiasAdd_3;box-2/BiasAdd_3;class_net/class-predict/separable_conv2d_4/depthwise;box-2/separable_conv2d_3;box-2/separable_conv2d_31,Reshape_1,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_3,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 386 ops Intermediate tensors: Reshape_3,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_2,Reshape_1,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_5,Reshape_6,Reshape,Reshape_4,box-2/separable_conv2d_3/depthwise;box-2/separable_conv2d_3/depthwise Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 385 ops Intermediate tensors: Reshape_3,Relu6_13;Relu6_13;box-1-bn-6/FusedBatchNormV3;box-1-bn-6/FusedBatchNormV3;box-1/BiasAdd/ReadVariableOp;box-1/BiasAdd/ReadVariableOp;box-1/BiasAdd_3;box-1/BiasAdd_3;class_net/class-predict/separable_conv2d_4/depthwise;box-1/separable_conv2d_3;box-1/separable_conv2d_31,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_2,Reshape_1,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_5,Reshape_6,Reshape,Reshape_4 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 384 ops Intermediate tensors: Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_1,Reshape_6,Reshape_5,box-1/separable_conv2d_3/depthwise;box-1/separable_conv2d_3/depthwise,Reshape,Reshape_4,Reshape_3 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 383 ops Intermediate tensors: Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_6,Reshape_5,Relu6_12;Relu6_12;box-0-bn-6/FusedBatchNormV3;box-0-bn-6/FusedBatchNormV3;box-0/BiasAdd/ReadVariableOp;box-0/BiasAdd/ReadVariableOp;box-0/BiasAdd_3;box-0/BiasAdd_3;class_net/class-predict/separable_conv2d_4/depthwise;box-0/separable_conv2d_3;box-0/separable_conv2d_31,Reshape_1,Reshape_4,Reshape,Reshape_3 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 382 ops Intermediate tensors: Reshape_5,Reshape,Reshape_4,box-0/separable_conv2d_3/depthwise;box-0/separable_conv2d_3/depthwise,Reshape_3,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_1,Reshape_6 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 381 ops Intermediate tensors: Reshape_5,Reshape,Reshape_4,Reshape_3,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_1,Reshape_6 Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 380 ops Intermediate tensors: Reshape_5,Reshape,Reshape_4,Reshape_3,Reshape_2,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_1,class_net/class-predict/BiasAdd_3;class_net/class-predict/separable_conv2d_4;class_net/class-predict/separable_conv2d_3;class_net/class-predict/BiasAdd/ReadVariableOp Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! Try to compile segment with 379 ops Intermediate tensors: Reshape_5,class_net/class-predict/separable_conv2d_3/depthwise,Reshape_1,cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/bn/FusedBatchNormV3;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd/ReadVariableOp;cell_4/fnode7/op_after_combine12/conv/BiasAdd;cell_4/fnode7/op_after_combine12/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_4/fnode7/op_after_combine12/conv/separable_conv2d;cell_4/fnode7/op_after_combine12/conv/separable_conv2d,Reshape_2,cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/bn/FusedBatchNormV3;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd/ReadVariableOp;cell_5/fnode6/op_after_combine11/conv/BiasAdd;cell_5/fnode6/op_after_combine11/conv/BiasAdd;class_net/class-predict/separable_conv2d_4/depthwise;cell_5/fnode6/op_after_combine11/conv/separable_conv2d;cell_5/fnode6/op_after_combine11/conv/separable_conv2d,Reshape_3,Reshape_4,Reshape Started a compilation timeout timer of 180 seconds. ERROR: Restored original execution plan after delegate application failure. Compilation failed: Compilation failed due to large activation tensors in model. Compilation child process completed within timeout period. Compilation failed! ```
hjonnala commented 2 years ago

Hi @ghbn8 can you please share the model that you are not to compile with the compiler. Thanks!

ghbn8 commented 2 years ago

Please find the EfficientDet Lite 3 with transfer learning attached.

082822_pm.zip

hjonnala commented 2 years ago

Reshape layers are not able to fit on edgeTPU memory. Please try the command below to compile the model. Thanks!

edgetpu_compiler -s -i "Reshape_1/shape,Reshape/shape" 082822_pm.tflite

ghbn8 commented 2 years ago

@hjonnala thank you, this resolved my issue -- with the referenced flags I am able to compile the model, and with the example below (from the Coral team) I am able to run inference with the model https://github.com/google-coral/pycoral/blob/master/examples/classify_image.py

Here is the command I ran to execute the example script above (referencing the labels.txt file): python3 examples/detect_image.py --model examples/082822_pm_edgetpu.tflite --labels test_data/labels.txt --input test_data/screenshot.jpg -o image_results.jpg

However, I am attempting to run inference with the example code snippet provided in my previous post (a minor modification to the code provided from the tflite team https://www.tensorflow.org/lite/inference_with_metadata/task_library/object_detector#step_2_using_the_model_3).

Here is the code snippet (similar to previous post, now includes argument for labels.txt file):

# Initialize the object detection model
base_options = core.BaseOptions(file_name=model, use_coral=enable_edgetpu, num_threads=num_threads)
detection_options = processor.DetectionOptions(display_names_locale='temp_code\labels.txt', max_results=3, score_threshold=0.3)
options = vision.ObjectDetectorOptions(base_options=base_options, detection_options=detection_options)

detector = vision.ObjectDetector.create_from_options(options)

My questions:

  1. Can the edgetpu model be compiled to include the necessary metadata (instead of needing to provide an external labels file)?
  2. If it cannot, should metadata be provided as a label.txt file?
  3. The code snippet I provided above is providing the error below, any idea why (I want to be sure it is not a problem with how I have compiled the model)?
Traceback (most recent call last):
  File "video_object_detection_run_on_pi.py", line 101, in <module>
    detector = vision.ObjectDetector.create_from_options(options)
  File "/home/pi/.local/lib/python3.7/site-packages/tensorflow_lite_support/python/task/vision/object_detector.py", line 83, in create_from_options
    options.detection_options)
ValueError: Object detection models require TFLite Model Metadata but none was found

API for providing the "display names locale" file (from tflite): https://www.tensorflow.org/lite/api_docs/python/tflite_support/task/processor/DetectionOptions

Full python script: video_object_detection_run_on_pi.zip

Thank you

hjonnala commented 2 years ago

Please check the tutorial at: https://www.youtube.com/watch?v=QqTLkw1IenI&list=RDCMUC0rqucBdTuFTjJiefW5t-IQ&index=35 and reach out to tflite support team for any further help: https://github.com/tensorflow/tflite-support. Thanks!

google-coral-bot[bot] commented 2 years ago

Are you satisfied with the resolution of your issue? Yes No

hjonnala commented 2 years ago

Please try the command below to map the max operations to EdgeTPU:

edgetpu_compiler -s -i "class_net/class-predict/BiasAdd;class_net/class-predict/separable_conv2d_4;class_net/class-predict/separable_conv2d;class_net/class-predict/BiasAdd/ReadVariableOp,box_net/box-predict/BiasAdd;box_net/box-predict/separable_conv2d_4;box_net/box-predict/separable_conv2d;box_net/box-predict/BiasAdd/ReadVariableOp" 082822_pm.tflite

https://github.com/google-coral/edgetpu/issues/656#issuecomment-1242413482