search

johnolafenwa / DeepStack

The World's Leading Cross Platform AI Engine for Edge Devices
Apache License 2.0
678 stars 107 forks source link

Benchmarking script #14

Closed robmarkcole closed 3 years ago

robmarkcole commented 4 years ago

I will happily create a benchmarking script, to evaluate speed of deepstack on different platforms. It would do a batch job of say 30 detections and present the runtime. I don’t want to reinvent the wheel, perhaps you have a script already?

johnolafenwa commented 4 years ago

We don't have a standard script yet for that. Please create it and make a PR to include it in benchmark directory.

robmarkcole commented 4 years ago

OK this will be a script which just posts every photo in a directory for processing. Wont include the photos, although we could have a standard set of photos in a seperate repo perhaps

robmarkcole commented 4 years ago

Created a notebook here (will breakout into script) and the results are in, with tflite being lightning fast, processing 15 images (obj detection):

Platform Speed (sec) Predictions
Mac Pro with deepstack (add version tag) 51.9 91
RPi4 with tflite-server 9.33 159

ps which is the 'benchmark directory'?

johnolafenwa commented 4 years ago

Interesting. Thanks for these @robmarkcole , i will push a nightly tag pending the setup of the cicd pipeline so you can benchmark against the new detection pipeline.

BTW, as i have not setup the benchmark directory, can you make a PR for these to include the benchmark scripts and we can review and make neccessary updates.

I am also curios, what are the specs for the MacBook Pro?

robmarkcole commented 4 years ago

Hardware Overview:

Model Name: MacBook Pro Model Identifier: MacBookPro14,1 Processor Name: Dual-Core Intel Core i5 Processor Speed: 2.3 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache (per Core): 256 KB L3 Cache: 4 MB Hyper-Threading Technology: Enabled Memory: 8 GB