Open tucan9389 opened 5 years ago
Anyone could contribute! Thanks. 😎
Model | Size | Minimum iOS Version |
---|---|---|
MobileNet | 17.1 | iOS11 |
MobileNetV2 | 24.7 | iOS11 |
MobileNetV2FP16 | 12.4 | iOS11.2 |
MobileNetV2Int8LUT | 6.3 | iOS12 |
Resnet50 | 102.6 | iOS11 |
Resnet50FP16 | 51.3 | iOS11.2 |
Resnet50Int8LUT | 25.7 | iOS12 |
Resnet50Headless | 94.4 | iOS11 |
SqueezeNet | 5 | iOS11 |
SqueezeNetFP16 | 2.5 | iOS11.2 |
SqueezeNetInt8LUT | 1.3 | iOS12 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
MobileNet | 16 | 18 | 19 | 33 | 43 | 35 |
MobileNetV2 | 21 | 18 | 21 | 46 | 64 | 53 |
MobileNetV2FP16 | 20 | 19 | 20 | 48 | 65 | 57 |
MobileNetV2Int8LUT | 21 | 21 | 20 | 53 | 64 | 53 |
Resnet50 | 27 | 25 | 26 | 61 | 78 | 63 |
Resnet50FP16 | 26 | 26 | 27 | 64 | 75 | 74 |
Resnet50Int8LUT | 27 | 25 | 26 | 60 | 77 | 75 |
Resnet50Headless | 18 | 13 | 18 | 36 | 54 | 53 |
SqueezeNet | 17 | 17 | 18 | 24 | 35 | 29 |
SqueezeNetFP16 | 17 | 17 | 18 | 24 | 36 | 29 |
SqueezeNetInt8LUT | 18 | 19 | 18 | 27 | 34 | 30 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
MobileNet | 18 | 20 | 21 | 35 | 46 | 37 |
MobileNetV2 | 23 | 21 | 23 | 48 | 67 | 55 |
MobileNetV2FP16 | 24 | 21 | 23 | 50 | 69 | 60 |
MobileNetV2Int8LUT | 23 | 23 | 22 | 55 | 67 | 56 |
Resnet50 | 30 | 28 | 29 | 64 | 82 | 66 |
Resnet50FP16 | 28 | 28 | 30 | 66 | 78 | 76 |
Resnet50Int8LUT | 29 | 28 | 28 | 63 | 80 | 78 |
Resnet50Headless | 19 | 13 | 18 | 36 | 54 | 54 |
SqueezeNet | 18 | 18 | 20 | 25 | 37 | 31 |
SqueezeNetFP16 | 18 | 18 | 19 | 26 | 38 | 31 |
SqueezeNetInt8LUT | 20 | 20 | 19 | 29 | 37 | 32 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
MobileNet | 23 | 23 | 23 | 23 | 20 | 23 |
MobileNetV2 | 23 | 23 | 23 | 20 | 13 | 17 |
MobileNetV2FP16 | 23 | 23 | 23 | 18 | 13 | 15 |
MobileNetV2Int8LUT | 23 | 23 | 23 | 16 | 13 | 16 |
Resnet50 | 23 | 23 | 23 | 14 | 11 | 14 |
Resnet50FP16 | 23 | 23 | 23 | 14 | 11 | 12 |
Resnet50Int8LUT | 23 | 23 | 23 | 15 | 11 | 12 |
Resnet50Headless | 23 | 23 | 23 | 23 | 16 | 17 |
SqueezeNet | 23 | 23 | 23 | 23 | 23 | 23 |
SqueezeNetFP16 | 23 | 23 | 23 | 23 | 22 | 23 |
SqueezeNetInt8LUT | 23 | 23 | 23 | 23 | 23 | 23 |
Model | Size | Minimum iOS Version |
---|---|---|
YOLOv3 | 248.4 | iOS12 |
YOLOv3FP16 | 124.2 | iOS12 |
YOLOv3Int8LUT | 62.2 | iOS12 |
YOLOv3Tiny | 35.5 | iOS12 |
YOLOv3TinyFP16 | 17.8 | iOS12 |
YOLOv3TinyInt8LUT | 8.9 | iOS12 |
MobileNetV2_SSDLite | 9.3 | iOS12 |
ObjectDetector | 63.7 | iOS12 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
YOLOv3 | 108 | 93 | 100 | 356 | 569 | 561 |
YOLOv3FP16 | 104 | 89 | 101 | 348 | 572 | 565 |
YOLOv3Int8LUT | 101 | 92 | 100 | 337 | 575 | 572 |
YOLOv3Tiny | 46 | 41 | 47 | 106 | 165 | 168 |
YOLOv3TinyFP16 | 51 | 41 | 44 | 103 | 165 | 167 |
YOLOv3TinyInt8LUT | 45 | 39 | 39 | 106 | 160 | 161 |
MobileNetV2_SSDLite | 31 | 31 | 31 | 109 | 141 | 134 |
ObjectDetector | 24 | 26 | 23 | 63 | 86 | 84 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
YOLOv3 | 108 | 93 | 100 | 357 | 569 | 561 |
YOLOv3FP16 | 104 | 89 | 101 | 348 | 572 | 565 |
YOLOv3Int8LUT | 102 | 92 | 102 | 338 | 576 | 573 |
YOLOv3Tiny | 46 | 42 | 48 | 106 | 166 | 169 |
YOLOv3TinyFP16 | 51 | 41 | 44 | 104 | 165 | 167 |
YOLOv3TinyInt8LUT | 45 | 39 | 40 | 107 | 160 | 161 |
MobileNetV2_SSDLite | 32 | 31 | 32 | 109 | 142 | 134 |
ObjectDetector | 25 | 26 | 23 | 64 | 87 | 85 |
Model vs. Device | XS | XS Max |
XR | X | 7+ | 7 |
---|---|---|---|---|---|---|
YOLOv3 | 8 | 10 | 9 | 2 | 1 | 1 |
YOLOv3FP16 | 9 | 10 | 8 | 2 | 1 | 1 |
YOLOv3Int8LUT | 9 | 10 | 9 | 2 | 1 | 1 |
YOLOv3Tiny | 21 | 22 | 20 | 8 | 5 | 5 |
YOLOv3TinyFP16 | 19 | 23 | 21 | 9 | 5 | 5 |
YOLOv3TinyInt8LUT | 21 | 24 | 23 | 8 | 5 | 5 |
MobileNetV2_SSDLite | 23 | 23 | 23 | 8 | 6 | 6 |
ObjectDetector | 23 | 23 | 24 | 14 | 10 | 11 |
Convert text to markdown table format
dict = {}
device_name = None
device_names = []
f = open("performance-classification.txt", 'r')
lines = f.readlines()
for line in lines:
componants = line.split()
# if len(componants) == 0:
if len(componants) == 0:
continue
if len(componants) != 5:
device_name = " ".join(componants)
device_names.append(device_name)
continue
else:
model_name = componants[0]
inference_time = componants[2]
total_time = componants[3]
fps = componants[4]
if model_name not in dict:
dict[model_name] = {}
dict[model_name][device_name] = {
"inference_time": inference_time,
"total_time": total_time,
"fps": fps
}
f.close()
def get_markdown_table(dict, colunm_name = "inference_time"):
result = ""
row_text = "| Model vs. Device | "
row_text = row_text + " | ".join(device_names)
row_text = row_text + " |"
result += row_text + "\n"
# print(row_text)
row_text = "| ----: | "
for i in range(len(device_names)):
row_text += ":----: | "
result += row_text + "\n"
# print(row_text)
for model_name in dict.keys():
row_text = "| " + model_name + " | "
for device_name in device_names:
row_text += dict[model_name][device_name][colunm_name] + " | "
result += row_text + "\n"
return result
# inference_time
# total_time
# fps
print()
print()
print()
print("## Object Detection")
print()
print("### Infernece Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="inference_time"))
print("### Total Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="total_time"))
print("### FPS")
print()
print(get_markdown_table(dict=dict, colunm_name="fps"))
XS Max
MobileNet : 18 20 23
MobileNetV2 : 18 21 23
MobileNetV2FP16 : 19 21 23
MobileNetV2Int8LUT : 21 23 23
Resnet50 : 25 28 23
Resnet50FP16 : 26 28 23
Resnet50Int8LUT : 25 28 23
Resnet50Headless : 13 13 23
SqueezeNet : 17 18 23
SqueezeNetFP16 : 17 18 23
SqueezeNetInt8LUT : 19 20 23
XR
MobileNet : 19 21 23
MobileNetV2 : 21 23 23
MobileNetV2FP16 : 20 23 23
MobileNetV2Int8LUT : 20 22 23
Resnet50 : 26 29 23
Resnet50FP16 : 27 30 23
Resnet50Int8LUT : 26 28 23
Resnet50Headless : 18 18 23
SqueezeNet : 18 20 23
SqueezeNetFP16 : 18 19 23
SqueezeNetInt8LUT : 18 19 23
7+
MobileNet : 43 46 20
MobileNetV2 : 64 67 13
MobileNetV2FP16 : 65 69 13
MobileNetV2Int8LUT : 64 67 13
Resnet50 : 78 82 11
Resnet50FP16 : 75 78 11
Resnet50Int8LUT : 77 80 11
Resnet50Headless : 54 54 16
SqueezeNet : 35 37 23
SqueezeNetFP16 : 36 38 22
SqueezeNetInt8LUT : 34 37 23
7
MobileNet : 35 37 23
MobileNetV2 : 53 55 17
MobileNetV2FP16 : 57 60 15
MobileNetV2Int8LUT : 53 56 16
Resnet50 : 63 66 14
Resnet50FP16 : 74 76 12
Resnet50Int8LUT : 75 78 12
Resnet50Headless : 53 54 17
SqueezeNet : 29 31 23
SqueezeNetFP16 : 29 31 23
SqueezeNetInt8LUT : 30 32 23
## Object Detection
### Infernece Time (ms)
| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- |
| MobileNet | 18 | 19 | 43 | 35 |
| MobileNetV2 | 18 | 21 | 64 | 53 |
| MobileNetV2FP16 | 19 | 20 | 65 | 57 |
| MobileNetV2Int8LUT | 21 | 20 | 64 | 53 |
| Resnet50 | 25 | 26 | 78 | 63 |
| Resnet50FP16 | 26 | 27 | 75 | 74 |
| Resnet50Int8LUT | 25 | 26 | 77 | 75 |
| Resnet50Headless | 13 | 18 | 54 | 53 |
| SqueezeNet | 17 | 18 | 35 | 29 |
| SqueezeNetFP16 | 17 | 18 | 36 | 29 |
| SqueezeNetInt8LUT | 19 | 18 | 34 | 30 |
### Total Time (ms)
| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- |
| MobileNet | 20 | 21 | 46 | 37 |
| MobileNetV2 | 21 | 23 | 67 | 55 |
| MobileNetV2FP16 | 21 | 23 | 69 | 60 |
| MobileNetV2Int8LUT | 23 | 22 | 67 | 56 |
| Resnet50 | 28 | 29 | 82 | 66 |
| Resnet50FP16 | 28 | 30 | 78 | 76 |
| Resnet50Int8LUT | 28 | 28 | 80 | 78 |
| Resnet50Headless | 13 | 18 | 54 | 54 |
| SqueezeNet | 18 | 20 | 37 | 31 |
| SqueezeNetFP16 | 18 | 19 | 38 | 31 |
| SqueezeNetInt8LUT | 20 | 19 | 37 | 32 |
### FPS
| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- |
| MobileNet | 23 | 23 | 20 | 23 |
| MobileNetV2 | 23 | 23 | 13 | 17 |
| MobileNetV2FP16 | 23 | 23 | 13 | 15 |
| MobileNetV2Int8LUT | 23 | 23 | 13 | 16 |
| Resnet50 | 23 | 23 | 11 | 14 |
| Resnet50FP16 | 23 | 23 | 11 | 12 |
| Resnet50Int8LUT | 23 | 23 | 11 | 12 |
| Resnet50Headless | 23 | 23 | 16 | 17 |
| SqueezeNet | 23 | 23 | 23 | 23 |
| SqueezeNetFP16 | 23 | 23 | 22 | 23 |
| SqueezeNetInt8LUT | 23 | 23 | 23 | 23 |
The results are weird. Maximum FPS is 23.. (Not 30) Need to test again. 😓
Models
Vision
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
TODO
Text
TODO
TODO
TODO
TODO
Related Links