Gstreamer to opencv

[thibault390]

Hi, I'm using coral dev board with ubuntu bionic image from balena and I tried to run the bodypix.py script.

I have a question is it possible to easily switch from gstreamer to opencv? Or is it possible to get back the final image as an input for another model like in preprocess?

[thibault390]

Please, if someone has an idea it could be really helpful for me

[Namburger]

Hi @thibault390, it's actually shouldn't be that hard to use opencv, considering the output processing is already handled by our code. Getting an opencv loop to grab and process the frame before feeding into the interpreter should as easy as this: https://gist.github.com/Namburger/f7e6c18af94ef3d6a70076a130eb1f7c#file-tflite_cv_objdetect-py-L78-L93

I suggest you give it a try first and check back if you ran into issues

[thibault390]

Hi Namburger,

Thanks for the reply and help. I used this programming technique to run posenet and blazeface. I know it's not really hard, but in this kind of script, we can easily separate the preprocess, inference and postprocess parts which is difficult to distinguish for me in your code with gstreamer.

For example, I have difficulties to take out the segmentation script part in your code to use as a postprocess part with opencv, and then keep the final result (segmented image) to use it as a dataset or input for a second model.

Moreover, I already tried to use your models in the edgetpu delegate for tensorflowLite but I'm facing to an issue because of unknowing custom operation:

interpreter.allocate_tensors() File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter.py", line 242, in allocate_tensors return self._interpreter.AllocateTensors() File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter_wrapper.py", line 115, in AllocateTensors return _interpreter_wrapper.InterpreterWrapper_AllocateTensors(self) RuntimeError: Encountered unresolved custom op: PosenetDecoderOp.Node number 1 (PosenetDecoderOp) failed to prepare.

[Namburger]

@thibault390 ahhhh, I see this error message:

RuntimeError: Encountered unresolved custom op: PosenetDecoderOp.Node number 1 (PosenetDecoderOp) failed to prepare.

Means that you need to register the posenet decoder ops, are you using our basic or a tflite interpreter?

[thibault390]

I'm using the basic one, I guess:

model="/home/project-bodypix/models/bodypix_mobilenet_v1_075_768_576_16_quant_edgetpu_decoder.tflite"

interpreter = Interpreter(model, experimental_delegates=[load_delegate('libedgetpu.so.1.0')])

[Namburger]

I see that you are using the tflite interpreter, for that you'll also need to register the posenet decoder.so, which requires building it from here (all of this is already handled in the edgetpu basic engine).

Then here is how you'd register it:

tpu = tflite.load_delegate('libedgetpu.so.1')
posenet = tflite.load_delegate('posenet_decoder.so')
interpreter = tflite.Interpreter('posenet_mobilenet_v1_075_353_481_quant_decoder.tflite'), 
                                               experimental_delegates=[tpu, posenet])

[Namburger]

Your best solution is to just use out PoseEngine, which wraps around the basic engine: https://github.com/google-coral/project-bodypix/blob/master/pose_engine.py

The pose engine has all the post processing, so it'd be a much easier solution then to re-impletementing it your self using the tflite API.

[thibault390]

Thanks, but I don't understand how can I build the requirements to get the posenet_decoder library for python, sorry:

posenet = tflite.load_delegate('posenet_decoder.so')

File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter.py", line 161, in load_delegate delegate = Delegate(library, options) File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter.py", line 90, in init self._library = ctypes.pydll.LoadLibrary(library) File "/usr/lib/python3.7/ctypes/init.py", line 442, in LoadLibrary return self._dlltype(name) File "/usr/lib/python3.7/ctypes/init.py", line 364, in init self._handle = _dlopen(self._name, mode) OSError: posenet_decoder.so: cannot open shared object file: No such file or directory Exception ignored in: <function Delegate.del at 0xffff95cf6950> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tflite_runtime/interpreter.py", line 125, in del if self._library is not None: AttributeError: 'Delegate' object has no attribute '_library'

[Namburger]

@thibault390 as I said, 1) if you just use our Posenet engine, then there is no need to build the posenet_decoder.so, and no need to rewrite any post processing code, it would be the most straight forward way to go for you.

2) If you want to continue using the the tflite interpreter, then you need to install bazel, clone that repo and build:

bazel build -c opt :posenet_decoder.so

Then install that to your linked library paths, similar to how libedgetpu.so was installed Source: https://github.com/google-coral/edgetpu/blob/master/src/cpp/posenet/BUILD#L92

[thibault390]

Ok thanks, I used the first option and OpenCV with the "DetectPosesInImage" function

inference_time, poses, heatmap, bodyparts = engine.DetectPosesInImage(frame)

and the segmentation looked good but too slow, so I tried by using "engine.ParseOutputs(data)" to switch between Gstreamer and OpenCV from bodypix_gl_imx.py script by using:

https://gist.github.com/patrickelectric/443645bb0fd6e71b34c504d20d475d5a

But I didn't get the segmentation result in the image, only simple image. Is it possible to get the segmented image from gstreamer to opencv?

[Namburger]

Well the outputs of the model doesn't give you a whole image, it gives these parameters: inference_time, poses, heatmap, bodyparts. So you'll need to draw those outputs back on the image or else you'll just display the original image

[mas-dse-dhpancha]

Hi thibault390 and Namburger, I'm also interested in doing the same for converting the bodypix.py from Gstreamer to OpenCV. From reading the comments, it seems that we need to use PoseEngine somehow in the opencv while loop. I'm a little confused on how we associate the image frame from the while loop. Do we need to modify PoseEngine somehow? Also, bodypix.py has a Callback class. Would that have to be modified somehow to be able to update image with our model?

[Naveen-Dodda]

@mas-dse-dhpancha

I am not sure if this is still in your interest to do so with opencv.

You need not modify PoseEngine, instead you can use PoseEngine as reference to understand how output tensors are post-processed. I would do following steps to run bodypix with opencv.

1. Start interpreter and feedin input image from opencv capture.
2. Post Process the output tensors from inference results ( you can take PoseEngine as inference)
3. Overlay results on input image before it is saved or displayed. Note: Expect increase in frame latency when you use opencv.

I am closing this for now, feel free to open if you have more questions.