Open murdockhou opened 4 years ago
Were you able to fully compile the model for the edge TPU? I was only able to get 39 supported ops + 8 that will run on the CPU for Yolo v4 Tiny.
@alexanderswerdlow I only tried this on cpu, don't know how it compatiable with TPU.
@alexanderswerdlow The total number of layers is only 47?
@hhk7734 Yep. I’ll double check tomorrow but I’m almost positive. The only significant change I made was leakyRelu -> Relu. I was able to get it running on the edge TPU but it runs quite horribly, compared to even yolo v3 tiny. The performance is fine with ~20ms per frame but the accuracy is very poor.
@alexanderswerdlow Can you share .tflite file? not _edgetpu.tflite I want to compare it with the layer I created.
Can't share the tflite but happy to share my config. It's basically the stock config just without leakyRelu, 416x416, the right filter value, etc.
Edit if you're curious, here's my edgetpu_compiler output:
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 599 ms.
Input model: v4.tflite
Input size: 5.72MiB
Output model: v4_edgetpu.tflite
Output size: 5.75MiB
On-chip memory used for caching model parameters: 4.82MiB
On-chip memory remaining for caching model parameters: 1.88MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 47
Operation log: v4_edgetpu.log
Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 39
Number of operations that will run on CPU: 8
Operator Count Status
PAD 2 Mapped to Edge TPU
CONCATENATION 1 More than one subgraph is not supported
CONCATENATION 6 Mapped to Edge TPU
SPLIT 3 Mapped to Edge TPU
MAX_POOL_2D 3 Mapped to Edge TPU
DEQUANTIZE 2 Operation is working on an unsupported data type
RESIZE_BILINEAR 1 Operation version not supported
QUANTIZE 6 Mapped to Edge TPU
QUANTIZE 1 Operation is otherwise supported, but not mapped due to some unspecified limitation
QUANTIZE 1 More than one subgraph is not supported
CONV_2D 19 Mapped to Edge TPU
CONV_2D 2 More than one subgraph is not supported
@alexanderswerdlow If you have time, I want you to see the below link. https://github.com/hhk7734/tensorflow-yolov4/blob/master/test/make_edgetpu_tflite.ipynb https://github.com/hhk7734/tensorflow-yolov4/issues/20
using input_size=416, inference time is ~60ms.
@alexanderswerdlow how did you manage to convert the yolo model?
I´m using the same config file as you and I managed to run the model with detect.py after quantizing it with
python convert_tflite.py --weights ./checkpoints/yolov4 --output ./checkpoints/yolov4-int8.tflite --quantize_mode int8
But if I try to convert it with
edgetpu_compiler ./checkpoints/yolov4-int8.tflite
I get
Edge TPU Compiler version 14.1.317412892
Invalid model: ./checkpoints/yolov4-int8.tflite
Model not quantized
I thought that running convert_tflite.py with --quantize_mode int8 takes care of this so I don´t really get what I´m missing.
@alexanderswerdlow @murdockhou I would like to know this too. I've been trying to convert the weights of YOLOv3 and v4 to tflite ( fully int 8 quantized ) using these helpful steps but I didn't get a result sadly. I get the exact same result as @ItsMeTheBee above. I am a beginner and I would like to use this for a project. Any guidance would be extremely appreciated
Tel me plese, how to use this int8 model in Android\ios project? I have error in my Android project: Cannot copy from a TensorFlowLite tensor (Identity) with shape [1, 13, 13, 993] to a Java object with shape [1, 2535, 4].
With float16 and float32 models run is ok.
No idea. It does not compile if used at the coral compiler, as it needs quantization even though I already did it
Okay a little something about that:
I was able to compile the model after some modifications in core/common.py and convert_tflite.py In core/common.py I changed leaky relu to relu.
In convert_tflite.py I tried using the old converter:
elif FLAGS.quantize_mode == 'int8':
converter.experimental_new_converter = False
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
converter.allow_custom_ops = True
converter.representative_dataset = representative_data_gen
With this the edgetpu_compiler can convert 17 ops:
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 340 ms.
Input model: checkpoints/yolov3-1800-int8.tflite
Input size: 8.42MiB
Output model: yolov3-1800-int8_edgetpu.tflite
Output size: 8.44MiB
On-chip memory used for caching model parameters: 1.84MiB
On-chip memory remaining for caching model parameters: 256.75KiB
Off-chip memory used for streaming uncached model parameters: 5.64MiB
Number of Edge TPU subgraphs: 1
Total number of operations: 135
Operation log: yolov3-1800-int8_edgetpu.log
Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 17
Number of operations that will run on CPU: 118
Operator Count Status
RESHAPE 18 More than one subgraph is not supported
ADD 6 More than one subgraph is not supported
MUL 18 More than one subgraph is not supported
DEQUANTIZE 8 Operation is working on an unsupported data type
CONV_2D 2 More than one subgraph is not supported
CONV_2D 11 Mapped to Edge TPU
CONCATENATION 9 More than one subgraph is not supported
STRIDED_SLICE 12 More than one subgraph is not supported
MAX_POOL_2D 6 Mapped to Edge TPU
LOGISTIC 12 More than one subgraph is not supported
QUANTIZE 17 More than one subgraph is not supported
QUANTIZE 7 Operation is otherwise supported, but not mapped due to some unspecified limitation
SPLIT_V 2 Operation not supported
RESIZE_BILINEAR 1 Operation version not supported
EXP 6 Operation is working on an unsupported data type
Now this was done for a tiny v3 model so it might not work for you, you can still give it a try though =) I´ve been hoping to get a better conversion performance so I´d still be glad about any hint in the right direction.
Wow great work there. Have you tested it in Coral? What are the FPS @ItsMeTheBee?
EDIT: I tried following your steps, but adding this line
converter.representative_dataset = representative_data_gen
caused an error about min/max tensors. Any idea for a workaround? I renamed all the leaky_relu instances to relu and then changed the tf.nn.leaky_relu(conv, alpha=something) to tf.nn.relu(conv) at common.py. Also I made the changes specified in convert_tflite.py, except the line I told you, to avoid the error. After that, I tried to compile the tflite model and the compiler stated the model was not quantized
HI @murdockhou
After I used your method, I successfully implemented the YOLOV4-int8 function and successfully detected the kite.jpg
But when using the yolov4-tiny-int8 version, the model was successfully converted, but an error occurred when using detect.py
The error is as follows:
Tensorflow Version:tensorflow-gpu=2.3.0 / 2.3.0rc0 (All test)
Code:
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
Hey @w840401 , Ive had this error too, it occurs because you're expecting more outputs than your model already has. Our model has 2 output branches while the code expects 3. I sadly do not know what the solution is but I'd like it too @murdockhou
I naively thought it was caused by a OS problem~
The same error occurred when the windows was changed to Linux 😆 please somebudy help us. @murdockhou
I tried using the coco dataset to convert to tflite, but I had empty min/max tensors. I really want to know how you got this to work @murdockhou . If as I said I omit the converter.representative_dataset = representative_data_gen line, I fail to quantize the model. The script runs, but if I compile with the coral model compiler, an error pops up saying The model is not quantized. Also, if I try to run the deepsort using this model, no bounding boxes are produced, which is weird. Could you help?
I tried using the coco dataset to convert to tflite, but I had empty min/max tensors. I really want to know how you got this to work @murdockhou . If as I said I omit the converter.representative_dataset = representative_data_gen line, I fail to quantize the model. The script runs, but if I compile with the coral model compiler, an error pops up saying The model is not quantized. Also, if I try to run the deepsort using this model, no bounding boxes are produced, which is weird. Could you help?
我成功解決您說的此問題,是因為val2017.txt 路徑問題 你重新執行coco_dataset 路徑對,應該就沒問題了
Hey @w840401 , I tried changing the path of val2017.txt, but it still has the same problem.
RuntimeError: Max and min for dynamic tensors should be recorded during calibration: Failed for tensor StatefulPartitionedCall/functional_1/zero_padding2d/Pad Empty min/max for tensor StatefulPartitionedCall/functional_1/zero_padding2d/Pad
Any ideas? My val2017.txt is at the same folder as the pictures, and points to them. For example
000000000139.jpg ..... for this specific picture.
Thanks for the help
@JimBratsos I don't think need at the same folder as the pictures Maybe can't points !
Quantification still has problems after this I did not choose to use tensorflow=2.3.0 I used tf-nighty~ Resource: https://pypi.org/project/tf-nightly/
My english is poor~ Sorry
When using the yolov4-tiny-int8 version, and run the model with detect.py
after quantizing it with
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
I got this error too.
output_tensors = decode(pred[2], input_size // 8, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
IndexError: list index out of range
Then, I added additional code in detect.py
, and I can successfully detect image now. The same as detectvideo.py
.
# add post process code here
bbox_tensors = []
prob_tensors = []
if FLAGS.tiny:
for i, fm in enumerate(pred):
if i == 0:
output_tensors = decode(pred[1], input_size // 16, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
else:
output_tensors = decode(pred[0], input_size // 32, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
bbox_tensors.append(output_tensors[0])
prob_tensors.append(output_tensors[1])
else:
for i, fm in enumerate(pred):
if i == 0:
output_tensors = decode(pred[2], input_size // 8, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
elif i == 1:
output_tensors = decode(pred[0], input_size // 16, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
else:
output_tensors = decode(pred[1], input_size // 32, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
bbox_tensors.append(output_tensors[0])
prob_tensors.append(output_tensors[1])
pred_bbox = tf.concat(bbox_tensors, axis=1)
pred_prob = tf.concat(prob_tensors, axis=1)
pred = (pred_bbox, pred_prob)
Although the yolov4-tiny-int8 model is successfully detected, it runs slowly, takes about 300ms for each image on CPU. Does anyone have suggestions about this?
@deep-rooteddz Thanks for sharing. I follow your step to modify my detect.py file and run, but the results went wrongs. It didn't get the similar results as the @murdockhou. Would you mind to show your inference results on kit.jpg? tks.
@deep-rooteddz Thanks for sharing. I follow your step to modify my detect.py file and run, but the results went wrongs. It didn't get the similar results as the @murdockhou. Would you mind to show your inference results on kit.jpg? tks.
@zero90169 I found that I can run @hunglc007 's code without problems, maybe he has fixed it. Of course, you can continue to use @murdockhou 's method. Using yolov4-tiny-int8.tflite to detect kite.jpg, the accuracy is the same.
If you don’t get the similar result, check config.py to confirm
YOLO.CLASSES = "./data/classes/coco.names"
TRAIN.ANNOT_PATH = "./data/dataset/val2017.txt"
TEST.ANNOT_PATH = "./data/dataset/val2017.txt"
And you may need to modify YOLO.ANCHORS_TINY to make it the same size as yolov4-tiny.weight(But not necessary, it will affect boundingbox size). e.g.
__C.YOLO.ANCHORS_TINY = [10,14, 23,27, 37,58, 81,82, 135,169, 344,319]
# __C.YOLO.ANCHORS_TINY = [23,27, 37,58, 81,82, 81,82, 135,169, 344,319]
Here is my yolov4-tiny-int8.tflite results on kite.jpg.
@deep-rooteddz Thanks for your kindly and quickly reply. I expected to get as many boxes as the kite.jpg which is shown by @murdockhou but my int8.tflite inference result on kite.jpg is more similar to yours. Maybe it is caused by "yolov4" and "yolov4-tiny". I will try the @hunglc007 's code again!
@deep-rooteddz Thanks. Your solution works for me, like you I experience significant slowness in relation to FP32, it is around 300MS instead of 100MS inference time, have you been able to resolve this? I use tiny yolo v4
When using the yolov4-tiny-int8 version, and run the model with
detect.py
after quantizing it with
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
I got this error too.
output_tensors = decode(pred[2], input_size // 8, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') IndexError: list index out of range
Then, I added additional code in
detect.py
, and I can successfully detect image now. The same asdetectvideo.py
.# add post process code here bbox_tensors = [] prob_tensors = [] if FLAGS.tiny: for i, fm in enumerate(pred): if i == 0: output_tensors = decode(pred[1], input_size // 16, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') else: output_tensors = decode(pred[0], input_size // 32, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') bbox_tensors.append(output_tensors[0]) prob_tensors.append(output_tensors[1]) else: for i, fm in enumerate(pred): if i == 0: output_tensors = decode(pred[2], input_size // 8, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') elif i == 1: output_tensors = decode(pred[0], input_size // 16, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') else: output_tensors = decode(pred[1], input_size // 32, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite') bbox_tensors.append(output_tensors[0]) prob_tensors.append(output_tensors[1]) pred_bbox = tf.concat(bbox_tensors, axis=1) pred_prob = tf.concat(prob_tensors, axis=1) pred = (pred_bbox, pred_prob)
Although the yolov4-tiny-int8 model is successfully detected, it runs slowly, takes about 300ms for each image on CPU. Does anyone have suggestions about this?
thanks, it works in my raspi 4, but why the int 8 model run slower than fp16 or 32?? I still not using any accelerator, full cpu. for int8 i got 1.3fps and fp16,32 almost similar at 2.3fps
did anyone solve this issue? my int8 converted model does not work with the script,
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
and also not working in android app too (fp16 converted model works though)
when I tested with int8 converted model, this error message is shown
ValueError: Shapes (1, 13, 13) and (1, 26, 26) are incompatible
@deep-rooteddz
Thanks, it works !
@murdockhou @deep-rooteddz @mipsan
Hi, everyone
I am using @hunglc007 's orginal detect.py to detect my own yolov4-tiny-int8.tflite.
Unfortunately, it shows the same error as @mhyeonsoo
My command: python detect.py --weights ./checkpoints/yolov4-tiny-224-int8.tflite --size 224 --model yolov4 --image ./data/person.jpg --framework tflite --tiny
and my environment: python 3.8 tf-nightly 2.6.0
Please give me some advice about the error, thks.
did anyone solve this issue? my int8 converted model does not work with the script,
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
and also not working in android app too (fp16 converted model works though)
when I tested with int8 converted model, this error message is shown
ValueError: Shapes (1, 13, 13) and (1, 26, 26) are incompatible
Have you solved it? I have encountered the same problem. Looking forward to your reply, thanks!
I'm also having problems trying to convert a YOLOv4 model to full int8 quantization, and I don't have an answer yet but I found a bug in the convert_tflite.py script that will always result in a model that is not fully int8 quantized. The lines:
elif FLAGS.quantize_mode == 'int8':
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
The second assignment to converter.target_spec.supported_ops supersedes the first that contains the int8 flag
did anyone solve this issue? my int8 converted model does not work with the script,
python detect.py --weights ./checkpoints/yolov4-tiny-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite --tiny
and also not working in android app too (fp16 converted model works though) when I tested with int8 converted model, this error message is shown
ValueError: Shapes (1, 13, 13) and (1, 26, 26) are incompatible
Have you solved it? I have encountered the same problem. Looking forward to your reply, thanks!
The reason is that you should add the post processing after invoking the model, since this was deleted from save_model.py script.
Hello,
I have tried to run the int8 model on the raspberry pi4. I got this error :
RuntimeError: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_selectNode number 79 (FlexFusedBatchNormV3) failed to prepare.
EDIT:
The same with float 16
thanks, it works in my raspi 4, but why the int 8 model run slower than fp16 or 32?? I still not using any accelerator, full cpu. for int8 i got 1.3fps and fp16,32 almost similar at 2.3fps
Hello, Can you confirm that the float16 model is faster on the raspberry pi ? Thanks
Hello,
While running this python convert_tflite.py --weights ./checkpoints/yolov4-416 --output ./checkpoints/yolov4-416-int8.tflite --quantize_mode int8
I'm getting the following error
[{'name': 'input_1', 'index': 0, 'shape': array([ 1, 416, 416, 3], dtype=int32), 'shape_signature': array([ -1, 416, 416, 3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
[{'name': 'Identity', 'index': 937, 'shape': array([1, 1, 1], dtype=int32), 'shape_signature': array([-1, -1, -1], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
Traceback (most recent call last):
File "convert_tflite.py", line 76, in
Can you please comment how to solve this error.
Hi, thanks for your nicely work. As you say,
Yolov4 and Yolov4-tiny int8 quantization have some issues. I will try to fix that.
I have a solution to fix this, and below is the code:First, when
Convert darknet weights to tensorflow
, we should not containpost process
insaved_model
:Then, run this command line convert this to
tflite-int8
:I disable this param
--dataset ./coco_dataset/coco/val207.txt
since I do not have a coco dataset, also, you should to comment line 43Finally, add some code in
detect.py
(actually they arepose process
code we comment in the first step):run
python detect.py --weights ./checkpoints/yolov4-416-int8.tflite --size 416 --model yolov4 --image ./data/kite.jpg --framework tflite
and we will get this result like this:Hope that can help you.