Use python3-flask and preload model into memory

[salekd]

The python3-flask template available via faas-cli template store pull can preload the model and engine having it ready to do inferences much much quicker.

[alexellis]

I've made the changes required and on my Intel NUC i5 I'm getting the following:

time curl -s 192.168.0.35:8081 -H "Content-Type: application/json" -d "$(echo -n '{"filename": "50-20171111115421-01.jpg", "image_data": "'; base64 res_28.jpg  | tr -d '\n'; echo '"}')"
{"detected_objects": [{"class": "car", "score": 0.8430330753326416}, {"class": "car", "score": 0.8217195272445679}, {"class": "car", "score": 0.7075631618499756}, {"class": "car", "score": 0.5339139103889465}, {"class": "car", "score": 0.4539302885532379}, {"class": "car", "score": 0.30341610312461853}, {"class": "car", "score": 0.2705892026424408}, {"class": "car", "score": 0.26135993003845215}, {"class": "car", "score": 0.2504091262817383}, {"class": "car", "score": 0.24646219611167908}, {"class": "car", "score": 0.2075640857219696}, {"class": "car", "score": 0.18780913949012756}, {"class": "car", "score": 0.16390538215637207}, {"class": "car", "score": 0.14526697993278503}, {"class": "car", "score": 0.14240741729736328}, {"class": "truck", "score": 0.12460744380950928}, {"class": "car", "score": 0.12042626738548279}, {"class": "car", "score": 0.1129477322101593}, {"class": "truck", "score": 0.10776272416114807}, {"class": "car", "score": 0.1022842526435852}, {"class": "car", "score": 0.10214415192604065}, {"class": "truck", "score": 0.0986972451210022}, {"class": "truck", "score": 0.09545770287513733}, {"class": "car", "score": 0.0871959924697876}, {"class": "car", "score": 0.08664137125015259}, {"class": "car", "score": 0.08298557996749878}, {"class": "car", "score": 0.08101525902748108}, {"class": "car", "score": 0.0807742178440094}, {"class": "car", "score": 0.07996904850006104}, {"class": "car", "score": 0.07880857586860657}, {"class": "car", "score": 0.07855868339538574}, {"class": "car", "score": 0.07829061150550842}, {"class": "car", "score": 0.07697266340255737}, {"class": "car", "score": 0.0766635537147522}, {"class": "car", "score": 0.07547017931938171}, {"class": "car", "score": 0.07342022657394409}, {"class": "car", "score": 0.07312142848968506}, {"class": "car", "score": 0.07304263114929199}, {"class": "truck", "score": 0.0729077160358429}, {"class": "car", "score": 0.07239335775375366}, {"class": "car", "score": 0.07210445404052734}, {"class": "car", "score": 0.06940868496894836}, {"class": "car", "score": 0.06688976287841797}, {"class": "car", "score": 0.06657156348228455}, {"class": "car", "score": 0.06492665410041809}, {"class": "car", "score": 0.0642390251159668}, {"class": "car", "score": 0.06393453478813171}, {"class": "car", "score": 0.0639316737651825}, {"class": "car", "score": 0.06220453977584839}, {"class": "car", "score": 0.061036139726638794}, {"class": "car", "score": 0.06064200401306152}, {"class": "car", "score": 0.05944448709487915}, {"class": "car", "score": 0.056974172592163086}, {"class": "fire hydrant", "score": 0.05568349361419678}, {"class": "car", "score": 0.05542007088661194}, {"class": "car", "score": 0.0549447238445282}, {"class": "car", "score": 0.05433526635169983}, {"class": "car", "score": 0.054201215505599976}, {"class": "car", "score": 0.05361980199813843}, {"class": "car", "score": 0.05322068929672241}, {"class": "car", "score": 0.052207112312316895}, {"class": "car", "score": 0.05159109830856323}, {"class": "person", "score": 0.051266878843307495}, {"class": "car", "score": 0.04980027675628662}, {"class": "truck", "score": 0.0493549108505249}, {"class": "car", "score": 0.04909563064575195}, {"class": "truck", "score": 0.04834529757499695}, {"class": "bench", "score": 0.047804296016693115}, {"class": "car", "score": 0.04766997694969177}, {"class": "bench", "score": 0.045695483684539795}, {"class": "bench", "score": 0.04519209265708923}, {"class": "car", "score": 0.04518309235572815}, {"class": "sports ball", "score": 0.044055283069610596}, {"class": "bench", "score": 0.04275193810462952}, {"class": "car", "score": 0.04174056649208069}, {"class": "car", "score": 0.041328996419906616}, {"class": "truck", "score": 0.041208237409591675}, {"class": "car", "score": 0.04118698835372925}, {"class": "car", "score": 0.04069721698760986}, {"class": "car", "score": 0.040068596601486206}, {"class": "truck", "score": 0.0399603545665741}, {"class": "car", "score": 0.03947994112968445}, {"class": "bus", "score": 0.039456576108932495}, {"class": "sports ball", "score": 0.039218246936798096}, {"class": "car", "score": 0.038990527391433716}, {"class": "car", "score": 0.038311511278152466}, {"class": "person", "score": 0.03815916180610657}, {"class": "bench", "score": 0.03806081414222717}, {"class": "car", "score": 0.03791326284408569}, {"class": "sports ball", "score": 0.037220388650894165}, {"class": "car", "score": 0.03694257140159607}, {"class": "car", "score": 0.03692391514778137}, {"class": "car", "score": 0.036576151847839355}, {"class": "car", "score": 0.036283135414123535}, {"class": "car", "score": 0.03619083762168884}, {"class": "car", "score": 0.03574511408805847}, {"class": "car", "score": 0.03544935584068298}, {"class": "bus", "score": 0.03510430455207825}, {"class": "truck", "score": 0.0349176824092865}, {"class": "person", "score": 0.034910887479782104}]}
real    0m0.427s
user    0m0.011s
sys 0m0.010s

That's around 0.4-0.5s per inference vs 5s.

[alexellis]

Have a look at my work here -> https://github.com/alexellis/openfaas-flask-mobilenet

[salekd]

Nice, that is quite a difference!