search

JasonDCox / ML-Mentorship-GovSchool

0 stars 0 forks source link

Write Results and Discussion Section #36

Closed gavinjalberghini closed 2 years ago

gavinjalberghini commented 2 years ago

Due Feb 28th

Acceptance Criteria:

brandonC1234 commented 2 years ago

Written Results and Discussion:

Results

Algorithm FPS mAP AP for Bella Memory usage (GB) GPU usage CPU usage
Yolo 3.8 0.497 0.8474 1.026 100% 30% (Average for cores)
tfLite 0.78 0.620 0.420 0.127 0% 25% (100% one core)
TensorRT 3.82 0.846 0.848 **4.9 100% 50%
*Tensorflow   0.751 0.384      

*only run on desktop **used Swap memory past 2GB

Discussion

FPS and processing usage TensorRT and YOLO both achieved nearly 4 fps, which is plenty for the application, while tfLite only achieved 0.78 fps. However, tfLite used significantly less resources, which may make it viable if the rest of the application needs a lot of resources to run.

Average Precisions In terms of performance between YOLO and TensorRT, both had high average precision for the unique dog (in this case Bella), which can be explained by how her breed was not in the dataset, making her the only dog of her class while the other breed classes had different looking dogs of the same breed. However, the models’ average precision varied much more for the other breeds with TensorRT maintaining a rather high AP and YOLO falling lower. What’s more surprising is that TensorRT beat Tensorflow by a large margin, which is very unexpected since they should lose a little performance in the optimization process. This result between Tensorflow, tfLite, and TensorRT, along with YOLO and TensorRT should be investigated in future research to determine if it's a repeatable occurrence or a lucky event.

Memory Usage The last notable result is the drastic difference in memory usage between tensorRT and YOLO, in which tensorRT utilized 4.9 GB of memory, resulting in it having to offload 2.9 GB to Swap memory. This is because the project is utilizing TF-TRT rather than dedicated TensorRT, which makes the program load the entirety of Tensorflow in the process, thus resulting in the large memory usage and a 20+ minute load time. Using dedicated TensorRT should have a more comparable memory to YOLO and similar prediction performance to the TF-TRT method utilized.