search

TexasInstruments / edgeai-gst-plugins

Repository to host GStreamer plugins for TI's EdgeAI class of devices
Other
18 stars 4 forks source link

Edge AI apps with rpi camera as input hangs after around 45 minutes #177

Closed ravi-rahul closed 2 years ago

ravi-rahul commented 2 years ago

Steps to reproduce

  1. get the tip of edge ai apps https://git.ti.com/git/edgeai/edge_ai_apps.git

  2. Edit the configs/image_classification.yaml file to set Rpi camera as input for the demo input0: source: /dev/video2 format: raw width: 1920 height: 1080 framerate: 30

    flow0: input: input0 models: [model0] outputs: [output0] mosaic:

  3. Run the app _cd /opt/edge_ai_apps/appspython _./apps_python.py ../config/imageclassification.yaml -nv

The app hangs after around 45 minutes This dose not happen when running the input pipeline standalone using gst-launch command

aniket-l commented 2 years ago

I ran the python demo with RPI camera input and object detection usecase (model2). I had added 2 identity elements (before and after the tioxisp plugin) and some prints to track the buffers and found that the ISP plugin was getting a buffer but not sending it out:

v4l2src device=/dev/video2 io-mode=5 ! video/x-bayer, width=1920, height=1080, format=rggb ! identity name=sniffer1 ! tiovxisp device=/dev/v4l-subdev2 dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin format-msb=7 ! video/x-raw, format=NV12 ! identity name=sniffer2 ! tiovxmultiscaler name=split_01 split_01. ! queue ! video/x-raw, width=512, height=512 ! tiovxdlpreproc data-type=10 channel-order=0 mean-0=123.675000 mean-1=116.280000 mean-2=103.530000 scale-0=0.017125 scale-1=0.017507 scale-2=0.017429 tensor-format=bgr out-pool-size=4 ! application/x-tensor-tiovx ! appsink name=pre_0 max-buffers=2 drop=true split_01. ! queue ! video/x-raw, width=1280, height=720 ! tiovxcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! appsink name=sen_0 max-buffers=2 drop=true

[GST SINK STR] appsrc format=GST_FORMAT_TIME is-live=true block=true do-timestamp=true name=post_0 ! tiovxcolorconvert ! video/x-raw,format=NV12, width=1280, height=720 ! queue ! mosaic_0.sink_0 appsrc format=GST_FORMAT_TIME block=true num-buffers=1 name=background_0 ! tiovxcolorconvert ! video/x-raw,format=NV12, width=1920, height=1080 ! queue ! mosaic_0.background tiovxmosaic name=mosaic_0 sink_0::startx=320 sink_0::starty=180 sink_0::width=1280 sink_0::height=720
! video/x-raw,format=NV12, width=1920, height=1080 ! kmssink sync=false driver-name=tidss

sniffer1 <gi.GstTIOVXRawImageBufferPool object at 0xffffad45d180 (GstTIOVXRawImageBufferPool at 0xffff6c03a130)> 2169322098978 65091 sniffer2 <gi.GstTIOVXImageBufferPool object at 0xffffad45d100 (GstTIOVXImageBufferPool at 0xffff7802cf30)> 2169322098978 65091 2022-01-12 13:33:37.459483 Got tensor from appsink sniffer1 <gi.GstTIOVXRawImageBufferPool object at 0xffffad45d100 (GstTIOVXRawImageBufferPool at 0xffff6c03a130)> 2169355403008 65092 sniffer2 <gi.GstTIOVXImageBufferPool object at 0xffffad45d340 (GstTIOVXImageBufferPool at 0xffff7802cf30)> 2169355403008 65092 2022-01-12 13:33:37.492662 Got tensor from appsink sniffer1 <gi.GstTIOVXRawImageBufferPool object at 0xffffad45d340 (GstTIOVXRawImageBufferPool at 0xffff6c03a130)> 2169388748063 65093 2022-01-12 13:33:37.525599 Got tensor from appsink 2022-01-12 13:33:37.558859 Got tensor from appsink [ERROR] Error pulling frame from GST Pipeline

As seen, buffer number 65093 is received by "sniffer1" but not "sniffer2". One more observation is that, atleast for a couple runs, this issue consistently occurs at about the same buffer number (60000+). Although I am not entirely sure.

m-herrera commented 2 years ago

@aniket-l @ravi-rahul I was able to reproduce this issue on our end and did some debugging.

The pipeline is indeed hanging during the ISP element processing. However, looking at the gstreamer log, there doesn't seem to be anything wrong going on.

From our testing, it seems that the execution hangs on the vxWaitGraph call here: L1450. The last logs we get are from the isp enqueuing parameters and scheduling the graph;1 however, it never reaches the parameter dequeuing section.

I also wanted to point out that the logs and the behavior for this issue look exactly like those from issue #166, is there any chance that both issues are related? Once again this hang issue happens on an element (ISP) that uses a VPAC target (VPAC_VISS) which is a common factor with the previous hang issue

This is the log I get from which I can tell that the vxWaitGraph is hanging.

[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'total time': 33.34 ms
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'framerate': 29.99 fps
0:44:07.177135023  1606     0x3c601e30 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert0> Dequeueing parameters
0:44:07.177662868  1606     0x3c601e30 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxdlpreproc0> Transferring handles
0:44:07.177737603  1606     0x3c601e30 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.177765315  1606     0x3c601e30 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.177788922  1606     0x3c601e30 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxdlpreproc0> Enqueueing parameters
0:44:07.177814223  1606     0x3c601e30 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxdlpreproc0> Processing graph
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Time for 'dl-inference':  3.68 ms (avg  3.49 ms)
0:44:07.181102990  1606     0x3c601e30 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxdlpreproc0> Dequeueing parameters
0:44:07.183960589  1606     0x3c6021e0 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert1> Dequeueing parameters
0:44:07.194670330  1606     0x3c601d90 LOG                 tiovxpad gsttiovxpad.c:381:gst_tiovx_pad_chain:<tiovxisp0:sink> Received a buffer for chaining
0:44:07.194739700  1606     0x3c601d90 LOG                tiovxsimo gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.194782047  1606     0x3c601d90 LOG                 tiovxpad gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.194802314  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1395:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing parameters
0:44:07.194820255  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1399:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.194844952  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1413:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.194865418  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.194884904  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.194900080  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1444:gst_tiovx_simo_process_graph:<tiovxisp0> Processing graph
0:44:07.194942673  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1450:gst_tiovx_simo_process_graph:<tiovxisp0> Graph Scheduled
0:44:07.195797734  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.195855358  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.195890341  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.195909377  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.195925378  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.195947219  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.195805440  1606     0x3c602230 LOG                tiovxsiso gsttiovxbufferutils.c:299:gst_tiovx_get_vx_array_from_buffer: TIOVX Image Meta was not found in buffer
0:44:07.197259746  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxcolorconvert2> Transferring handles
0:44:07.197356572  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.197382134  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.197400860  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Enqueueing parameters
0:44:07.197426092  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Processing graph
0:44:07.199196569  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
0:44:07.208632501  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Dequeueing parameters
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'total time': 33.34 ms
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'framerate': 29.99 fps
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Time for 'dl-inference':  3.24 ms (avg  3.49 ms)
0:44:07.229169366  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.229170941  1606     0x3c602230 LOG                tiovxsiso gsttiovxbufferutils.c:299:gst_tiovx_get_vx_array_from_buffer: TIOVX Image Meta was not found in buffer
0:44:07.229239341  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.229290564  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.229320341  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.229336882  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.229369004  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.230380901  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxcolorconvert2> Transferring handles
0:44:07.230452536  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.230481817  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.230500314  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Enqueueing parameters
0:44:07.230523930  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Processing graph
0:44:07.232380063  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
0:44:07.241706428  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Dequeueing parameters
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'total time': 33.34 ms
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'framerate': 29.99 fps
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Time for 'dl-inference':  3.21 ms (avg  3.49 ms)
0:44:07.262514945  1606     0x3c602230 LOG                tiovxsiso gsttiovxbufferutils.c:299:gst_tiovx_get_vx_array_from_buffer: TIOVX Image Meta was not found in buffer
0:44:07.262509465  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.262934038  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.263393618  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.263680732  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.263759922  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.263677522  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxcolorconvert2> Transferring handles
0:44:07.263851218  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.263886705  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.263919868  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Enqueueing parameters
0:44:07.263948859  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Processing graph
0:44:07.263826876  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.266910545  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
0:44:07.275155213  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Dequeueing parameters
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'total time': 33.34 ms
[UTILS] [TVM-CL-3410-gluoncv-mxnet-mobv2] Metric 'framerate': 29.99 fps
0:44:07.295785778  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.295789508  1606     0x3c602230 LOG                tiovxsiso gsttiovxbufferutils.c:299:gst_tiovx_get_vx_array_from_buffer: TIOVX Image Meta was not found in buffer
0:44:07.295836751  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.296085693  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.296510201  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.296837767  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.296912092  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxcolorconvert2> Transferring handles
0:44:07.297015079  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.297052396  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.297084529  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Enqueueing parameters
0:44:07.297125436  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Processing graph
0:44:07.296982302  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.300129949  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
0:44:07.308347916  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Dequeueing parameters
0:44:07.329107174  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.329151007  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.329108114  1606     0x3c602230 LOG                tiovxsiso gsttiovxbufferutils.c:299:gst_tiovx_get_vx_array_from_buffer: TIOVX Image Meta was not found in buffer
0:44:07.329195355  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.329388333  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.329786864  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.330068062  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.330229768  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:469:gst_tiovx_siso_transform:<tiovxcolorconvert2> Transferring handles
0:44:07.330277066  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 1
0:44:07.330299758  1606     0x3c602230 LOG                tiovxsiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.330316339  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:864:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Enqueueing parameters
0:44:07.330337225  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:883:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Processing graph
0:44:07.333357194  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
0:44:07.341561675  1606     0x3c602230 LOG                tiovxsiso gsttiovxsiso.c:896:gst_tiovx_siso_process_graph:<tiovxcolorconvert2> Dequeueing parameters
0:44:07.362429541  1606     0x3c6022d0 DEBUG              tiovxmiso gsttiovxmiso.c:545:gst_tiovx_miso_aggregate:<mosaic_0> TIOVX Miso aggregate
0:44:07.362470083  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.362528757  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxutils.c:269:gst_tiovx_transfer_handle: Number of planes to transfer: 2
0:44:07.362547283  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:577:gst_tiovx_miso_aggregate:<mosaic_0:sink_1> ignoring EOS and re-using previous buffer
0:44:07.362573525  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:456:gst_tiovx_miso_process_graph:<mosaic_0> Enqueueing parameters
0:44:07.362601552  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:486:gst_tiovx_miso_process_graph:<mosaic_0> Processing graph
0:44:07.365507053  1606     0x3c6022d0 LOG                tiovxmiso gsttiovxmiso.c:499:gst_tiovx_miso_process_graph:<mosaic_0> Dequeueing parameters
[ERROR] Error pulling tensor from GST Pipeline

It is clearer looking only at the ISP output:

0:44:07.128144417  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1395:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing parameters
0:44:07.128161818  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1399:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.128186509  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1413:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.128206431  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.128224257  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.128239238  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1444:gst_tiovx_simo_process_graph:<tiovxisp0> Processing graph
0:44:07.128280621  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1450:gst_tiovx_simo_process_graph:<tiovxisp0> Graph Scheduled
0:44:07.132506559  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1459:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing parameters
0:44:07.132536331  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1462:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.132602471  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1476:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.132625297  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1493:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.132643608  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1493:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.132837031  1606     0x3c601d90 LOG                 tiovxisp gsttiovxisp.c:1440:gst_tiovx_isp_postprocess:<tiovxisp0> Exposure time output from TI 2A library: 33333
0:44:07.132855627  1606     0x3c601d90 LOG                 tiovxisp gsttiovxisp.c:1442:gst_tiovx_isp_postprocess:<tiovxisp0> Analog gain output from TI 2A library: 8192
0:44:07.161339361  1606     0x3c601d90 LOG                 tiovxpad gsttiovxpad.c:381:gst_tiovx_pad_chain:<tiovxisp0:sink> Received a buffer for chaining
0:44:07.161476290  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1395:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing parameters
0:44:07.161493567  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1399:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.161517913  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1413:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.161538239  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.161556876  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.161573402  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1444:gst_tiovx_simo_process_graph:<tiovxisp0> Processing graph
0:44:07.161617900  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1450:gst_tiovx_simo_process_graph:<tiovxisp0> Graph Scheduled
0:44:07.165807891  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1459:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing parameters
0:44:07.165835693  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1462:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.165858695  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1476:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.165880001  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1493:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.165898122  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1493:gst_tiovx_simo_process_graph:<tiovxisp0> Dequeueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.166111911  1606     0x3c601d90 LOG                 tiovxisp gsttiovxisp.c:1440:gst_tiovx_isp_postprocess:<tiovxisp0> Exposure time output from TI 2A library: 33333
0:44:07.166130292  1606     0x3c601d90 LOG                 tiovxisp gsttiovxisp.c:1442:gst_tiovx_isp_postprocess:<tiovxisp0> Analog gain output from TI 2A library: 8192
0:44:07.194670330  1606     0x3c601d90 LOG                 tiovxpad gsttiovxpad.c:381:gst_tiovx_pad_chain:<tiovxisp0:sink> Received a buffer for chaining
0:44:07.194802314  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1395:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing parameters
0:44:07.194820255  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1399:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing input array of refs: 0x3c3908f8  with graph id: 0
0:44:07.194844952  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1413:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing output array of refs: 0x3c390c60     with graph id: 1
0:44:07.194865418  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3906e8  with graph id: 2
0:44:07.194884904  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1430:gst_tiovx_simo_process_graph:<tiovxisp0> Enqueueing queueable array of refs: 0x3c3907f0  with graph id: 3
0:44:07.194900080  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1444:gst_tiovx_simo_process_graph:<tiovxisp0> Processing graph
0:44:07.194942673  1606     0x3c601d90 LOG                tiovxsimo gsttiovxsimo.c:1450:gst_tiovx_simo_process_graph:<tiovxisp0> Graph Scheduled

I added this extra log, just to make sure that the problem was not with the schedule call.

diff --git a/gst-libs/gst/tiovx/gsttiovxsimo.c b/gst-libs/gst/tiovx/gsttiovxsimo.c
index 4738464..56f972b 100644
--- a/gst-libs/gst/tiovx/gsttiovxsimo.c
+++ b/gst-libs/gst/tiovx/gsttiovxsimo.c
@@ -1447,6 +1447,7 @@ gst_tiovx_simo_process_graph (GstTIOVXSimo * self)
     GST_ERROR_OBJECT (self, "Schedule graph failed %" G_GINT32_FORMAT, status);
     goto exit;
   }
+  GST_LOG_OBJECT (self, "Graph Scheduled");
   status = vxWaitGraph (priv->graph);
   if (VX_SUCCESS != status) {
     GST_ERROR_OBJECT (self, "Wait graph failed %" G_GINT32_FORMAT, status);
ravi-rahul commented 2 years ago

This was related to rpmsg issue we have a fix for this in SDK