running from DVP camera

zhen8838 / K210_Yolo_framework

Yolo v3 framework base on tensorflow, support multiple models, multiple datasets, any number of output layers, any number of anchors, model prune, and portable model to K210 !

MIT License

249 stars 73 forks source link

running from DVP camera #8

Open kervel opened 4 years ago

kervel commented 4 years ago

Hello,

We tried to get the C code running on K210 maix bit, using the latest version of the standalone SDK. I see your main.c contains code for DVP but its commented. so we tried to bring it to life but so far not succesfull.

I have some questions:

we get 0.2 FPS on mobile V1 with dm=0.75 .. is this realistic ? the animated gif seemed much faster. we had to hack the latest standalone sdk to get it working (remove an assert in dvp.c !is_memory_cache)
we don't get meaningful results (always the same rectangle). might be a bug in our trials.

i guess you used this repo mainly for creating a model rather than using it as a base for inference on K210. what would be the best practises for doing inference based on the models here ?

/cc @gilleskap

Greetings, Frank

krishnak commented 4 years ago

Did you create your KModel with latest nncase? it is bound to have operations that are performed on CPU which might increase the inference time. I can offer you commercial consultancy if you are interested

zhen8838 commented 4 years ago

0.2fps may be because you didn't rewrite region layer.c. the real-time demo detection code I showed was rewritten by me.

The output is the same, You should refer to the official real-time Yolo detection demo

kervel commented 4 years ago

Hello @zhen8838 and @krishnak, thanks for your reply ... i'll check but i'm not sure to which code you refer (official real-time demo). could you give me a pointer ?

as for commercial consultancy, thanks for the offer. i'm not sure if that will be possible in my project but at least i need to prove feasibility first.

zhen8838 commented 4 years ago

demo link: https://github.com/kendryte/nncase/tree/master/examples/20classes_yolo

kervel commented 4 years ago

Hello @zhen8838, thanks for the link! we didn't look at that one because it seems it contains also a preloaded image (image.c) and no image recognition on the DVP signal. or do you want to indicate this is a good base for starting a conversion to DVP based image recognition ?

zhen8838 commented 4 years ago

which board did you use ? my code and kendryte demo code only support kendryte KD233 board.

The DVP driver will auto convert rgb565 image to rbg888 format and put into kpu memory, so you just set kpu memory address when board init.

image.c just contain height, width information.

kervel commented 4 years ago

hi @zhen8838 sorry for the confusion. I didn't use it on a board yet, i just tried to find out if the demo is a real time demo (i'll probably use a maix bit in the first stage)

so i looked at https://github.com/kendryte/nncase/blob/master/examples/20classes_yolo/k210/kpu_20classes_example/main.c#L248 and i saw the kmodel uses a reference to gImage_image which is defined in https://raw.githubusercontent.com/kendryte/nncase/master/examples/20classes_yolo/k210/kpu_20classes_example/image.c as containing pixel data.

i compared this to https://github.com/kendryte/nncase/blob/master/examples/facedetect_landmark/k210/facedetect_landmark_example/main.c which does contain code to get image data from the camera. Should i try to combine the two ?

thanks!

zhen8838 commented 4 years ago

yes. The facedetect_landmark is realtime demo. The yolo demo just infer once, you can custom image array.