search

Freescale / gstreamer-imx

GStreamer 1.0 plugins for i.MX platforms
Other
184 stars 127 forks source link

imxvpudec problem: "Could not push input frame data into decoder: invalid call" #293

Closed guillerodriguez closed 1 year ago

guillerodriguez commented 3 years ago

I am trying to test a basic RTSP / H.264 pipeline as follows:

gst-launch-1.0 rtspsrc location=rtsp://192.168.0.241 ! rtph264depay ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

This fails with the following error messages, and nothing shown onscreen:

[...]
0:00:02.346060667  0x75003750 ERROR            imxvpuapi imxvpuapi2_imx6_coda.c:1949:imx_vpu_api_dec_push_encoded_frame: tried to push an encoded frame before a previous one was decoded
0:00:02.346159667  0x75003750 ERROR            imxvpudec gstimxvpudec.c:629:gst_imx_vpu_dec_handle_frame:<imxvpudech264-0> could not push input frame data into decoder: invalid call
0:00:02.346554667  0x75003750 ERROR            imxvpuapi imxvpuapi2_imx6_coda.c:1949:imx_vpu_api_dec_push_encoded_frame: tried to push an encoded frame before a previous one was decoded
0:00:02.346615000  0x75003750 ERROR            imxvpudec gstimxvpudec.c:629:gst_imx_vpu_dec_handle_frame:<imxvpudech264-0> could not push input frame data into decoder: invalid call
0:00:02.346849667  0x75003750 ERROR            imxvpuapi imxvpuapi2_imx6_coda.c:1949:imx_vpu_api_dec_push_encoded_frame: tried to push an encoded frame before a previous one was decoded
0:00:02.346897333  0x75003750 ERROR            imxvpudec gstimxvpudec.c:629:gst_imx_vpu_dec_handle_frame:<imxvpudech264-0> could not push input frame data into decoder: invalid call
[...continues...]

Same happens with imxvpudec_mpeg4 although surprisingly imxvpudec_jpeg works.

Replacing imxg2dvideosink with fbdevsink doesn't help. Adding some queues to the pipeline doesn't help either.

Versions:

Any hint what could be happening here ?

dv1 commented 3 years ago

This is indeed odd. Does this happen with any RTSP h264 signal? How did you generate the signal? Did you use VLC or gst-rtsp-server?

guillerodriguez commented 3 years ago

This is with a live RTSP stream coming from an IP door station. The same setup was working fine with the 0.x series (been using 0.13.1 until now).

I will try with other stream sources and report back.

guillerodriguez commented 3 years ago

@dv1 Have tried other RTSP live sources (IP cameras and door stations), same problem. All of these work fine with the 0.x series.

Also tried using gst-rtsp-server:

./test-launch  --gst-debug=3 '( v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,width=320,height=240 ! x264enc tune="zerolatency" ! video/x-h264, profile=baseline ! rtph264pay name=pay0 pt=96 )'

The above works fine with 0.13.1. With 2.0.0 it seems to be hit and miss, either it works fine, or it starts dumping "tried to push an encoded frame before a previous one was decoded" + "could not push input frame data into decoder: invalid call" error messages and nothing is shown onscreen. From my tests it seems to work like 50% of the time...

Any hints?

guillerodriguez commented 3 years ago

@dv1 Any other tests I can run? Anything else I can do?

dv1 commented 3 years ago

@guillerodriguez Unfortunately I am dealing with something else at the moment. I'll try to make some room for this in a couple of days.

guillerodriguez commented 3 years ago

@dv1 Any news (not meant to push)? If there's anything I can do to help please let me know..

dv1 commented 3 years ago

I could not reproduce this yet. Can you reproduce this on a Sabre SD board? Also, can you re-run this with the GST_DEBUG environment variable set to *vpu*:9 and attach the logs?

guillerodriguez commented 3 years ago

I could not reproduce this yet. Can you reproduce this on a Sabre SD board? Also, can you re-run this with the GST_DEBUG environment variable set to *vpu*:9 and attach the logs?

Hi, sorry for the delay, I did not have the test setup at hand. I don't have a Sabre SD board, I am doing all tests on a Variscite VAR-SOM-SOLO/DUAL + eval kit.

However I can re-run with GST_DEBUG set to *vpu*:9 and attach the logs.

Test setup:

RTSP source using gst-rtsp-server:

./test-launch  --gst-debug=3 '( v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,width=320,height=240
! x264enc tune="zerolatency" ! video/x-h264, profile=baseline ! rtph264pay name=pay0 pt=96 )'

Gstreamer pipeline:

GST_DEBUG_FILE=/tmp/gst.log GST_DEBUG=*vpu*:9 gst-launch-1.0 rtspsrc location=rtsp://192.168.0.155:8554/test
! rtph264depay ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

As described in my earlier comment, this sometimes work, sometimes doesn't. Seems to be hit and miss.

I am attaching the logs of one of the cases where it did NOT work. The symptoms:

Log file: gst_streaming_err.txt

guillerodriguez commented 3 years ago

@dv1 Any news? Anything else I can do to help?

dv1 commented 3 years ago

@guillerodriguez This indicates something odd with the h264 frames. It would be useful to have a short example bitstream. Can you run this pipeline for a few seconds?

gst-launch-1.0 rtspsrc location=rtsp://192.168.0.155:8554/test ! rtph264depay ! h264parse ! "video/x-raw,stream-format=byte-stream" ! filesink location=test.h264

Then check if the file can reproduce that error by running this pipeline:

GST_DEBUG_FILE=/tmp/gst.log GST_DEBUG=*vpu*:9 gst-launch-1.0 filesrc location=test.h264 ! "video/x-raw,stream-format=byte-stream" ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

If so, please attach the test.h264 file here, along with the logfile.

guillerodriguez commented 3 years ago

@dv1 Looks like the pipeline won't run as the h264parse element cannot handle video/x-raw caps:

WARNING: erroneous pipeline: could not link h264parse0 to filesink0, h264parse0 can't handle caps video/x-raw, stream-format=(string)byte-stream

However I did the following:

  1. Record the stream as H.264 video:

    gst-launch-1.0 rtspsrc location=rtsp://192.168.0.155:8554/test ! rtph264depay ! h264parse ! "video/x-h264, stream-format=byte-stream" ! filesink location=test.h264

  2. Verify that I get the same "invalid call" errors if we try to reproduce the recorded stream:

    GST_DEBUG_FILE=/tmp/gst.log GST_DEBUG=*vpu*:9 gst-launch-1.0 filesrc location=test.h264 ! "video/x-h264,stream-format=byte-stream" ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

  3. Verify that the same stream can be reproduced on non-iMX6 hardware (a standard PC) without issues:

    gst-launch-1.0 filesrc location=test.h264 ! "video/x-h264, stream-format=byte-stream" ! h264parse ! avdec_h264 ! autovideosink

I am attaching the (zipped) test.h264 file and the gst.log from step 2 above. Hoperfully this makes it easier to troubleshoot the issue. Please let me know if you would like me to run any other tests.

dv1 commented 3 years ago

@guillerodriguez Finally got a chance to check it out. Interestingly, trying to play this with vlc, ffplay, gst-play, mpv produces different initial results. Something is wrong with the first few frames. I suspect the bitstream is subtly broken, leading to undefined behavior. Still verifying.

guillerodriguez commented 2 years ago

@dv1 Any updates on this? Anything I can do to help, please let me know

dv1 commented 2 years ago

@guillerodriguez I have to take care of other issues atm, but I can get back to this in a few days Sorry for the delay.

guillerodriguez commented 2 years ago

Hi @dv1

@guillerodriguez I have to take care of other issues atm, but I can get back to this in a few days Sorry for the delay.

Just wanted to check the status of this. Can I help in any way to make this move forward?

dv1 commented 2 years ago

I ran this as the sender:

./test-launch  --gst-debug=3 '( v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,width=320,height=240 ! x264enc tune="zerolatency" ! video/x-h264, profile=baseline ! rtph264pay name=pay0 pt=96 )'

and this on an imx6dl sabre SD:

GST_DEBUG=2,*imx*:4 gst-launch-1.0 rtspsrc location=rtsp://172.16.1.14:8554/test ! rtph264depay ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

and I do not get any errors.

However, this is on Yocto/OE Dunfell, the newest git master versions of gstreamer-imx, and libimxvpuapi 2.2.0. Can you retry with those? And do you use Buildroot or Yocto?

Also, I use GStreamer 1.18.2 on the Sabre board and 1.18.5 on the host. Dunfell has the NXP fork of GStreamer 1.16 by default, so I add the Dunfell branch of https://github.com/OSSystems/meta-gstreamer1.0 and added this to my OE local.conf to ensure that the NXP fork is not picked by bitbake:

PREFERRED_VERSION_gstreamer1.0 = ""
PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
PREFERRED_VERSION_gstreamer1.0-libav = ""

Can you try that? Perhaps it is a mix of gstreamer-imx and GStreamer issues. I suspect that what you saw is caused by incomplete frames, which can occur over RTSP.

I also tried to first start another RTSP client on the host before running the rtspsrc pipeline on the Sabre board to provoke incomplete frames (since the sender pipeline starts running because of the first client). Still no problems.

Also, perhaps the issue you reported was caused by a bug that was fixed by https://github.com/Freescale/libimxvpuapi/commit/1827394eb30479df535d04d659bef463051cb877 ?

guillerodriguez commented 2 years ago

Hello @dv1 ,

I ran this as the sender:

./test-launch  --gst-debug=3 '( v4l2src device=/dev/video0 ! videoconvert ! video/x-raw,width=320,height=240 ! x264enc tune="zerolatency" ! video/x-h264, profile=baseline ! rtph264pay name=pay0 pt=96 )'

and this on an imx6dl sabre SD:

GST_DEBUG=2,*imx*:4 gst-launch-1.0 rtspsrc location=rtsp://172.16.1.14:8554/test ! rtph264depay ! h264parse ! imxvpudec_h264 ! imxg2dvideosink clear-at-null=true

and I do not get any errors.

However, this is on Yocto/OE Dunfell, the newest git master versions of gstreamer-imx, and libimxvpuapi 2.2.0. Can you retry with those? And do you use Buildroot or Yocto?

Retried with those, using Yocto and with the same versions. The issue persists.

Also, I use GStreamer 1.18.2 on the Sabre board and 1.18.5 on the host. Dunfell has the NXP fork of GStreamer 1.16 by default, so I add the Dunfell branch of https://github.com/OSSystems/meta-gstreamer1.0 and added this to my OE local.conf to ensure that the NXP fork is not picked by bitbake:

PREFERRED_VERSION_gstreamer1.0 = ""
PREFERRED_VERSION_gstreamer1.0-plugins-base = ""
PREFERRED_VERSION_gstreamer1.0-plugins-good = ""
PREFERRED_VERSION_gstreamer1.0-plugins-bad = ""
PREFERRED_VERSION_gstreamer1.0-libav = ""

Can you try that? Perhaps it is a mix of gstreamer-imx and GStreamer issues. I suspect that what you saw is caused by incomplete frames, which can occur over RTSP.

Tried. The issue persists.

In my earlier comment (https://github.com/Freescale/gstreamer-imx/issues/293#issuecomment-948514338) I provided specific step by step instructions to reproduce the problem, and attached a recorded video session. Can you try that instead of your v4l2src? This way we are both testing the same thing and it is 100% reproducible.

dv1 commented 2 years ago

Indeed the issue persists. The main cause is this:

imxvpuapi imxvpuapi2_imx6_coda.c:2348:imx_vpu_api_dec_decode: not enough input data was available

Something is off with the incoming h264 data - h264parse is not producing whole frames, or at least the first frame is incomplete. Both v1 and v2 actually expect complete encoded frames (hence the "parsed: true" in the sink caps). In v1, this incomplete frame was sort of ignored, that is, its PTS/DTS/context values were overwritten by followup frames. In v2, doing that produces an error. Not sure yet how to fix this - if I turn off the check, I get working playback, but the first second or so is just a green frame (not unexpected since the data seems corrupted). Did you get a green frame at first with v1?

dv1 commented 2 years ago

@guillerodriguez Can you try applying this patch on libimxvpuapi (not gstreamer-imx!) and retrying? 0001-imx6-coda-Skip-incomplete-frames-instead-of-just-rep.patch.gz

guillerodriguez commented 2 years ago

[...] if I turn off the check, I get working playback, but the first second or so is just a green frame (not unexpected since the data seems corrupted). Did you get a green frame at first with v1?

Yes, with v1 I get a green frame for the first second or so, then after a while I see the video playback correctly.

guillerodriguez commented 2 years ago

Hi,

@guillerodriguez Can you try applying this patch on libimxvpuapi (not gstreamer-imx!) and retrying? 0001-imx6-coda-Skip-incomplete-frames-instead-of-just-rep.patch.gz

Testing this now, preliminary results show that the issue is gone (will need to do more thorough tests).

Is it possible to backport this fix to libimxvpuapi 0.x ?

dv1 commented 2 years ago

Is it possible to backport this fix to libimxvpuapi 0.x ?

No, this is not feasible. v1 has the problem that you must use a function that blocks until enough framebuffers have been returned to the decoder. There's no way to add framebuffers on the fly. This blocking function causes a lot of problems in more complex GStreamer pipelines where all framebuffers can end up all over the place and can't be returned. Normally, one would then just add more framebuffers, but in v1, that's not doable, as said, so the only tool in that version is to allocate more framebuffers than necessary during initialization. That extra number is arbitrary and needs to be chosen manually, usually by trial and error. v2 solved this in various way depending on the VPU (imx6 vs imx8). I'd have to backport a fair amount of v2 code to v1, which makes no sense. Add to that the fact that I do not have the capacity to maintain both v1 and v2, and you can see why this just can't be done.

dv1 commented 1 year ago

Closing this, as it seems completed. Reopen if it isn't please.