search

techyian / MMALSharp

C# wrapper to Broadcom's MMAL with an API to the Raspberry Pi camera.
MIT License
195 stars 33 forks source link

Error decoding over network stream #119

Closed tottaka closed 4 years ago

tottaka commented 4 years ago

Hi, I am trying to record and encode video from one raspberry pi camera and send it via tcp network stream to another raspberry pi where I can decode the video stream and show the video frames on screen. I am getting an error when I decode one frame, the decoder seems to freeze and doesn't accept any more data. I have attached my code in a gist here: https://gist.github.com/tottaka/0223e943e3deeb7a8c730d43f2f4fcb0

any help is greatly appreciated, also I am using the dev branch

tottaka commented 4 years ago

this is the output on the sending end (the rpi that is recording with camera) image

and this is the output I get on the receiving end (that should display the live camera feed): image

you can notice that both the receiving and sending ends freeze up after one frame is decoded on the receiving end, and i am not sure why.

techyian commented 4 years ago

Hi,

I really like what you're doing here. I suspect it probably is the decoder component which is having problems - I wonder if adding the constraint <FileEncodeOutputPort> to the ConfigureOutputPort call here may help improve things? You will just need to pass the output port number to that overloaded method, in your case it'll be 0. I know the name of that port could potentially be a bit better, but it has been made specifically to handle MMAL_EVENT_FORMAT_CHANGED events which are raised by the encoder/decoder components when being fed data from non-proprietary communication (i.e. not connected to another component). Please see the examples starting from here for some extra guidance.

techyian commented 4 years ago

Also if you could turn on verbose logging using MMALCameraConfig.Debug = true and send me the logs generated then that might also shed some light as to what's happening.

tottaka commented 4 years ago

Hey I tried adding the FileEncodeOutputPort constraint and now the output handler doesn't call the process method at all it seems like, it now freezes exactly when EOF = true instead of one call afterwards if that makes any sense. I would also like to add that when I change the output handler to a FileStreamCaptureHandler, it works as expected and I can play back the raw camera feed from the file using VLC. Also I added MMALCameraConfig.Debug = true and I don't see any extra output, is it supposed to generate a file or something?

techyian commented 4 years ago

Hi,

There's a few things here. Firstly, I've raised and committed #121 as the input stream into the decoder was completing too early - the buffer header size between the encoder's output port (server-side) and the decoder's input port (client-side) weren't the same so the EOF check was being set to true too early.

Next, your configuration is incorrect between your components. Your MMALVideoDecoder is expecting H.264 at 1280x720 at a framerate of 25 fps, see here. However, your MMALVideoEncoder on the server is sending video at 1920x1080 (the default for video) at a framerate of 30 fps. This is causing the video decoder to get confused and is simply blocking.

Before here, add the following 2 config changes to reflect what the decoder is expecting:

MMALCameraConfig.VideoResolution = new Resolution(1280, 720);
MMALCameraConfig.VideoFramerate = new MMAL_RATIONAL_T(25, 1);

You will also need to get latest on the dev branch for those changes committed to #121.

That seems to have sorted it locally for me using your source code. I would keep an eye on the performance though with you using TCP - if you experience any issues, maybe try UDP instead.

tottaka commented 4 years ago

Awesome! I was considering switching the video streaming to UDP as well, I was just trying to get something working to see if it would actually work haha. Great work by the way, thanks a ton!!

techyian commented 4 years ago

No problem :) glad it's working for you. Are you happy for the ticket to be closed? v0.6 should hopefully be released next week (if all goes to plan) so you'll be able to grab it from NuGet instead of github.

tottaka commented 4 years ago

Yup you are all good to close, I have implemented the changes you suggested and everything is working great. Thanks again!

techyian commented 4 years ago

Fantastic, thanks.

techyian commented 4 years ago

For some reason your comment hasn't appeared but I did receive the email notification:

Hi, sorry to reopen this.. It may still be part of the same issue? or maybe I have it configured wrong, I can't figure it out... The bytes that are input into the decoder on the rpi receiving side are then output, but all the bytes in the output are zero... I'm not sure what is going on here. This screenshot is the first 30 or so bytes of every input/output process event on the receiving(decoding) side, hope it helps.

I will try and see why this is happening this evening.

tottaka commented 4 years ago

Hey I think it was because I had the output setup for rgb16 when the camera was giving it i420. I have changed this and the data in the output is no longer all zeros, so I believe it is working I just can't check yet because I have no way of viewing the frames until I figure out how I can display i420 with opengl or convert it to jpeg or something. I deleted my question since I figured after changing the pixel format and the output looked right then it was actually working.

techyian commented 4 years ago

Ahh I see! No problem - I just got the email notification this morning and couldn't see the comment on here. What is your end goal here so I can understand what it is you're trying to do?

tottaka commented 4 years ago

Hey so basically I'm just trying to record live video from the camera with one rpi, send it over to an rpi zero which is connected to my TV, and then decode the video frames and display them on the tv with opengl. I'm using Dear ImGUI for 2d drawing with opengl and I have that part working I just can't figure out how to display the raw i420 frames yet. I looked around and saw some people using glsl to convert the i420 to rgb and display it that way, which is what I'm trying to do now. Hopefully it works. If you have any advice as to how you would go about this I would love to hear it!

techyian commented 4 years ago

The Video Decoder component does allow RGB565 to be set on the output port. I've just tried it and it does seem to produce valid output using the following port configuration:

MMALPortConfig outputPortConfig = new MMALPortConfig(MMALEncoding.RGB16, MMALEncoding.RGB16, 1280, 720, 25, 0, 1300000, true, null);

Does that produce the output you're looking for?

tottaka commented 4 years ago

When I do that, the output gives me all zero bytes like I was saying in the post I deleted...

tottaka commented 4 years ago

image That message is printed each time a new frame is decoded, and the first 32 bytes of the frame data (which is the working data from the OutputCaptureHandler) are displayed for testing.. as you can see they are all zero.

The decoding side is setup like so:

MMALPortConfig inputPortConfig = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, (int)Resolution.X, (int)Resolution.Y, Framerate, Quality, Bitrate, true, null);

MMALPortConfig outputPortConfig = new MMALPortConfig(MMALEncoding.RGB16, MMALEncoding.RGB16, (int)Resolution.X, (int)Resolution.Y, Framerate, Quality, Bitrate, true, null);

And I am sure the bitrate and quality are matching on the encoding/decoding side.

tottaka commented 4 years ago

Here is the encoding side if it helps at all image GenericStreamCaptureHandler: https://gist.github.com/tottaka/a8727cc99c29e18a4b3ca0c0f89060ee

techyian commented 4 years ago

I have made a Gist of what's working for me. It's pretty much the same as what you sent me but I'm using RGB16 (RGB565) on the Video Decoder's output port with a FileEncodeOutputPort generic constraint.

https://gist.github.com/techyian/ff4ff5018c6d4031af201489b6c5d186

techyian commented 4 years ago

The Video Decoder will buffer for a short time before it starts outputting any data. I'm not sure if that's the issue you're experiencing?

tottaka commented 4 years ago

Have you actually tried to verify the output though? Because it is outputting bytes, but they are all zero for me for some reason? If you have verified that it's outputting correctly, how did you do that so I can try it as well?

tottaka commented 4 years ago

Actually now it seems to be working, apparently I had to set the ReceiveBufferSize = 33554432; // 32 MB for the receiving end.

tottaka commented 4 years ago

So I think my problem was that when I try to save the decoded frame data to a file, for some reason it is only writing zeros to the file, and even though the frame data length is like 1.75MB, the file write operation completes instantly and the resulting file is the correct size but contains only zeros.. Do have any idea why this might be?

tottaka commented 4 years ago

Hey just want to let you know that you can close this. Everything is working now, there was a bug on my side so your project is working fine now. Thanks for everything!!

techyian commented 4 years ago

That's great, thanks for letting me know!