Cannot play Http stream with imxvpudec

[emustafa]

Hi,

I am trying to use imx plugins to play a http stream in humming board. Here is my pipeline that works when I use avdec_h264:

gst-launch-1.0 souphttpsrc location=http://x.x.x.x:xxxx/ ! flvdemux ! h264parse ! avdec_h264 ! imxipuvideotransform ! imxeglvivsink

However, when I replace avdec_h264 with imxvpudec then the stream does not start. It just hangs.

I am not sure if this is related to the vpu restriction that you mentioned under decoder notes.

I would appreciate any help.

Thanks.

[isaacrj]

As you may already know, if you do not include any queues in your pipeline, you are forcing all elements to perform their duties sequentially in the same thread. In our experience, this often leads to pipeline lockups.

Can you give this pipeline a try? gst-launch-1.0 souphttpsrc location=... ! queue ! flvdemux ! queue ! imxvpudec ! queue ! imxeglvivsink

If it does work, you can then try to remove the queues one by one until your pipeline no longer works. We usually just leave them there, though, as they don't impact the memory footprint very much and they tend to make the pipelines much more resilient to hiccups.

If it does not work, please, give us a sample of your flv file so we can reproduce the issue.

[emustafa]

Thank you for your response. I didn't know that without queues you get one thread.

So I tried

gst-launch-1.0 souphttpsrc location=... ! queue ! flvdemux ! queue ! imxvpudec ! queue ! imxeglvivsink

But it fails with the following error message:

ERROR: from element /GstPipeline:pipeline0/GstSoupHTTPSrc:souphttpsrc0: Internal data flow error.

I think I have to use h264parse after flvdemux beucase in that case I don't get any errors. So when I run

gst-launch-1.0 souphttpsrc location=... ! queue ! flvdemux ! queue ! h264parse ! queue ! imxvpudec ! queue ! imxeglvivsink

Then I am still getting the same result. It just hangs. The output is the following:

Setting pipeline to PAUSED ... [INFO] Product Info: i.MX6Q/D/S GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications. Pipeline is PREROLLING ... [INFO] bitstreamMode 1, chromaInterleave 0, mapType 0, tiled2LinearEnable 0

I also tried to remove queue in various locations but the same result. I think maybe it is because of the way flv is setup?

So what I am doing is I am screening capturing with vlc in a linux machine and streaming it using http. The stream is encoded using x264. And I am trying to play that stream in hm. Without hardware accelerated decoding the latency is too high and cpu usage is too high as well. So it is so important to get imxvpudec working with the stream. If you want to replicate the issue then the vlc command is the following. Actually at this point vlc parameters does not really matter. I want imxvpudec to work with simple x264 encoded vlc stream. I can work on optimization later.

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{preset=ultrafast,tune=zerolatency,intra-refresh=1,keyint=5,vbv-maxrate=6000,vbv-bufsize=3000,profile=main,nobframes}, vb=6250, scale=0.1,acodec=none}:http{mux=ffmpeg{mux=flv},dest=:8080}'

Note the scale there. When it is small (0.1 - 0.3) then avdec_h264 works fine. However, I need at least 0.8 scale which avdec_h264 can't handle.

[emustafa]

I actually have an update.

So I tried the following vlc command:

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{}, vb=6250, scale=1}:http{mux=ffmpeg{mux=flv},dest=:8080}'

without any x264 parameter changes. It worked with the queues. That is great to see. I guess I can now work on getting right vlc parameters.

I would like want to know what I did wrong in the previous vlc command.

Thanks.

[dv1]

Interesting. The pipeline should even work without queues, unless the full pipeline also connects an audio sink to flvdemux, for example - in that case, you do need queues. What was your initial FLV source? I would like to look at this. Or can you produce a log with the original (queue-less) pipeline that hangs with imxvpudec? Run it with GST_DEBUG=2,*imx*:5,*vpu*:9 please.

[emustafa]

Oh yes. You are right. The pipeline works without queues as well. I just didn't try at first.

[dv1]

Okay, nice. Still, can you try to produce a log with the original pipeline (with imxvpudec, not avdec_h264) and stream? Perhaps there is a corner case that isn't being addressed at the moment.

[emustafa]

Alright I ran the following command:

GST_DEBUG=2,imx:5,vpu:9 gst-launch-1.0 souphttpsrc location=http://192.168.0.106:8080/ ! flvdemux ! h264parse ! imxvpudec ! imxeglvivsink 1> vpudec.log 2> vpudec.log

The log file is in the link:

https://drive.google.com/file/d/0B2vLfCIAnJXiM2lydmNXTzRrcHM/view?usp=sharing

[dv1]

This is odd. I see "VPU_DecDecodeBuf returns: 841" all over the place. 0x40 is this ominous "mosaic frame", which is scarcely document in the VPU wrapper. Would it be possible to make a small capture of this FLV stream for me to test with? This never occurred before.

[emustafa]

So you want me to capture screen to a file and upload it?

[dv1]

Just run gst-launch-1.0 -e souphttpsrc location=http://192.168.0.106:8080/ ! filesink location=capture.flv (the -e switch is important here!) for a little while and then press Ctrl+C. The result should recreate the behavior you saw.

[emustafa]

So I ran the gst command you provided and here is the captured file https://drive.google.com/file/d/0B2vLfCIAnJXiR2RYdDJNakZVUWM/view?usp=sharing. It is 35 seconds file but starts from 14-15. I guess it is because of the latency. Latency is too high.

[dv1]

Hmm, the VPU does not seem to like this one. I get 0x821 return codes all over. 0x20 means "dropped". Interestingly, avdec_h264 can play it. Not sure what is going on. How do you produce the video?

[emustafa]

I am running this vlc command:

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{preset=ultrafast,tune=zerolatency,intra-refresh=1,keyint=5,vbv-maxrate=6000,vbv-bufsize=3000,profile=main,nobframes}, vb=6250, scale=0.1,acodec=none}:http{mux=ffmpeg{mux=flv},dest=:8080}'

Then as you suggested I ran:

gst-launch-1.0 -e souphttpsrc location=http://192.168.0.106:8080/ ! filesink location=capture.flv

That is how I got capture.flv.

But I as said when I use this command:

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{}, vb=6250, scale=0.6}:http{mux=ffmpeg{mux=flv},dest=:8080}'

Then it plays fine.

So I am guessing when I change parameters it changes the frames in certain way so that imxvpudec can't handle?

[dv1]

Ahh, try to set intra-refresh to 0 in your original vlc command. What happens then?

[emustafa]

By setting intra-refresh to 0, I am getting the same thing.

I think the best way to find which parameter is causing problem is starting from this command

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{}, vb=6250, scale=0.6}:http{mux=ffmpeg{mux=flv},dest=:8080}'

and add parameters until imxvpudec does not work. Then we will have a better idea which element is causing problem. I will let you know in a bit.

[isaacrj]

We are using imxvpudec to decode live flv h264 video from rtmp streams, and we did have quite a lot of problems when the streams were encoded with x264 using both zerolatency and ultrafast. Finally we settled with zerolatency and superfast to get it going...

Hope this helps.

[emustafa]

Thank you for your suggestion. I tried superfast and it actually worked. It started but the latency was so high. More than 20 seconds. What other tricks can I do to decrease the latency? Can you help?

[dv1]

try profile=baseline instead of profile=main

[emustafa]

I guess the best way, as I said, is to start from venc=x264{} and add parameters until it does not work. I will you know guys.

[isaacrj]

We are using gstreamer to encode here. We are encoding a live webcam using just x264enc bitrate=512 key-fram-int=15 tune=zerolatency speed-preset=superfast and our latency is well below 250ms.

From other projects, we know that with VLC it is almost impossible to get under 500ms due to its internal buffering sheme, but 20 seconds is a lot. I think you should take a look at the http side of the vlc command and make sure that its output buffers are set to a minimum.

[emustafa]

In our project, we are using raspberry pi right now. We want to move to a different board and we thought humming board would be a better choice.

In raspberry pi, with omxplayer and vlc we are getting < 500 ms latency. I think with hummingboard we should be able to get similar same latency or less latency.

The other thing, we should handle 1080p videos as well. I am guessing with live webcam you are not capturing high quality video. That helps as well because encoding/decoding part takes less time for low quality videos.

I am stuck so bad. I need to get this done as soon as possible.

[isaacrj]

You are right, we are using low resolution videos, but I don't think resolution is your problem here: if you are encoding realtime video at 25fps, your encoder has 40ms (mean) to encode each frame, or it will start dropping them. The same is true for your decoder. So either your hardware can cope with the resolution without dropping frames or it can not.

I tend to think that initial delay problems are related to the encoder needing a lot of frames before sending out its first encoded packet (which is true for certain configurations of H264, where frames depend on each other to be decoded, but a 20 seconds delay equals to a 500 frames buffer which I think is an awful lot for this hypothesis), the latency of the screengrabber itself or latency induced by any output buffer present on your http server.

If it is an option, you could try to use gstreamer in the encoding side too, as you will get much more control this way.

Sorry I can't offer better advice...

[emustafa]

Thank you for your suggestions. I will actually try to use gstreamer for encoding part and see if it makes difference.

[emustafa]

Hi again,

I am still trying to get vlc working. So as I said without any x264 parameters changes imxvpudec works. I am trying to change parameters so that the latency is reasonable and the video is smooth.

Right now, my vlc command is the following:

vlc -vvv -I dummy screen:// --screen-fps=25 --sout='#transcode{vcodec=h264,venc=x264{preset=superfast,tune=zerolatency,keyint=5,profile=baseline},scale=1}:http{mux=ffmpeg{mux=flv},dest=:8080}'

And my gstreamer pipeline is the following:

gst-launch-1.0 souphttpsrc location=... ! flvdemux ! h264parse ! queue ! imxvpudec ! imxeglvivsink

However, when I play a video even with low quality I am getting this warning from gstreamer:

WARNING: from element /GstPipeline:pipeline0/GstImxEglVivSink:imxeglvivsink0: A lot of buffers are being dropped. Additional debug info: gstbasesink.c(2794): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstImxEglVivSink:imxeglvivsink0: There may be a timestamping problem, or this computer is too slow.

and the video is not smooth and in fact it is not watchable. This only happens when I watch video. Do you think it is related to vlc or gstreamer? Because when I decrease the scale to 0.5 then it is fine. Do you think it is the vlc being late in encoding? Or if it is the imxvpudec then what should I do?

[isaacrj]

It could be a all process in a thread issue. Try adding a queue between ipxvpudec and imxeglvivsink.

When building gstreamer pipelines i.MX6, I strongly suggest to add a queue between each pair of elements and leave them there while you tweak the other elements. Once you get a working pipeline, you can remove the queues one by one to improve memory footprint.

If adding queues does not help, can you run 'dmesg' and see if you are getting some Warnings/Errors about the GPU not being able to handle the throughput or the VPU not having enough memory to do its work? We did face this issues in the past when trying to decode 1080p video at native resolution (that is, rendering it to a 1920x1080 framebuffer or window)

[emustafa]

Unfortunately, adding queue everywhere actually makes things worse. However, I added queue between imxvpudec and imxeglvivsink and I think it helped a bit.

When I do dmesg the only thing related to VPU is the following:

[12283.480346] mxc_vpu 2040000.vpu: VPU blocking: timeout. [12283.986255] mxc_vpu 2040000.vpu: VPU blocking: timeout. [12284.492281] mxc_vpu 2040000.vpu: VPU blocking: timeout. [12284.998281] mxc_vpu 2040000.vpu: VPU blocking: timeout. [12285.504220] mxc_vpu 2040000.vpu: VPU blocking: timeout. [12306.522490] mxc_vpu 2040000.vpu: VPU timeout during release

There was nothing related to GPU.

[isaacrj]

We should wait for dv1 to throw some light at this one. From the dmesg log, I'm guessing that imxvpudec is late decoding some frames or that it can not decode them at all. These frames are dropped and you get warnings from imxeglvivsink because in a realtime pipeline, usually the sink element is the one that notices that frames are missing and therefore is the element that posts the warnings even when it is not involved in the framedropping

[emustafa]

Yeah, you are right. We should wait for dv1.

I am not sure about vpu thing though.

There is another thing I suspect is happening is the wifi chip on the hm. Maybe it is late in downloading the frames. I will connect it to the router through ethernet and see if it makes difference.

[emustafa]

Alright! I was right! I plugged in ethernet and now I am streaming 1080p video with around 0.5 s latency!

I am so happy right now! I will tweak vlc parameters to derease the latency. I am not going to close the issue because imxvpudec does not work with all x264 parameters.

Thank you guys for your help. This made my day.

[dv1]

Alright then. I take it that x264enc output with ultrafast+zerolatency leads to problems, while superfast+zerolatency does not. I'll use that as a test case, but would close this issue, since it appears it it working for you now, correct?

[emustafa]

Yes it is working for me now. However, it would be great to see imxvpudec working with all kind of x264 parameters. I will close the issue.