MV10
commented
3 years ago Annnnd I think everything I wrote above is probably wrong, lol.
Apparently NativeConnectionCallback is also invoked for hardware-based inputs/outputs, which I didn't expect, so if I manage to get it right, we can compare pure native overhead (crazy fast) to managed callback overhead. This is my output below, that first one is the encoder (so 481 calls to NativeConnectionCallback, most of which never took even 1ms, 241 calls to the resizer InputCallback which makes sense at 24FPS for 10 seconds). The second set of timings is the resizer, it doesn't seem to ever be called at all...? (If you read this from email, I originally misinterpreted those.)
How the test is set up:
using (var capture = new VideoStreamCaptureHandler(rawPathname))
using (var resizer = new MMALIspComponent())
using (var encoder = new MMALVideoEncoder())
{
var camConfig = new MMALPortConfig(MMALEncoding.OPAQUE, MMALEncoding.I420);
var rawConfig = new MMALPortConfig(MMALEncoding.RGB24, MMALEncoding.RGB24, width: 640, height: 480);
var h264Config = new MMALPortConfig(MMALEncoding.H264, MMALEncoding.I420, quality: 10, bitrate: MMALVideoEncoder.MaxBitrateLevel4);
resizer.ConfigureInputPort(camConfig, cam.Camera.VideoPort, null);
encoder.ConfigureInputPort(rawConfig, null);
resizer.ConfigureOutputPort(rawConfig, null);
encoder.ConfigureOutputPort(h264Config, capture);
var connection = resizer.Outputs[0].ConnectTo(encoder, useCallback: true);
connection.RegisterCallbackHandler(new RawFrameCallbackFilter(connection));
cam.Camera.VideoPort.ConnectTo(resizer);
await Task.Delay(2000);
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(totalSeconds));
await cam.ProcessAsync(cam.Camera.VideoPort, cts.Token);
}Changes to that if-else block in MMALSharp sans timing and counters (which didn't break the resizer, I noticed):
protected virtual int NativeConnectionCallback(MMAL_CONNECTION_T* connection)
{
if (MMALCameraConfig.Debug)
{
MMALLog.Logger.LogDebug("Inside native connection callback");
}
var queue = new MMALQueueImpl(connection->Queue);
var bufferImpl = queue.GetBuffer();
if (bufferImpl.CheckState())
{
if (MMALCameraConfig.Debug)
{
bufferImpl.PrintProperties();
}
if (bufferImpl.Length > 0)
{
this.CallbackHandler.InputCallback(bufferImpl);
}
this.InputPort.SendBuffer(bufferImpl);
}
else
{
MMALLog.Logger.LogInformation("Connection callback input buffer could not be obtained");
}
queue = new MMALQueueImpl(connection->Pool->Queue);
bufferImpl = queue.GetBuffer();
if (bufferImpl.CheckState())
{
if (MMALCameraConfig.Debug)
{
bufferImpl.PrintProperties();
}
if (bufferImpl.Length > 0)
{
this.CallbackHandler.OutputCallback(bufferImpl);
}
this.OutputPort.SendBuffer(bufferImpl);
}
else
{
MMALLog.Logger.LogInformation("Connection callback output buffer could not be obtained");
}
return (int)connection->Flags;
}And finally, my do-nothing Callback handler:
public class RawFrameCallbackFilter : ConnectionCallbackHandler
{
public RawFrameCallbackFilter(IConnection connection)
: base(connection)
{ }
public override void InputCallback(IBuffer buffer)
{
base.InputCallback(buffer);
}
public override void OutputCallback(IBuffer buffer)
{
base.OutputCallback(buffer);
}
}I'll keep poking at it, but as we say, ELI5 ... Explain Like I'm 5. 😁
techyian
In #172 you suggested that a Connection callback handler might allow me to write a "filter" in the middle of the pipeline, such as:
camera -> resizer -> (FILTER) -> h.264 encoder -> video capture handler -> h.264 file
Given your warning this is new and somewhat uncharted territory, I thought I'd open a dedicated issue.
You mentioned that
MMALConnectionImpl.NativeConnectionCallbackwill only invokeInputCallbackbecause of that if-else block. Could the fix be as simple as pulling the output callback out of the else block? In other words, should it check the input queue and invoke the callback if there is data, then do the same for the output queue and callback?Assuming both input and output are being invoked, is this basically the flow for a derived Callback handler?
InputCallbackto be invokedGetBufferdata, readAssertPropertyetc.OutputCallbackto be invokedReadIntoBufferto populate any locally stored buffer dataI suppose I'm a little confused about what the port
SendBuffercalls do with the buffer data inNativeConnectionCallback. Is that because this type of component can have multiple inputs and outputs? (Can anything have multiple inputs? I don't think I've seen that in any examples.)Since you mentioned you aren't sure of the exact performance hit, I thought I would start with a simple pass-through and hack in something to display the raw ms/frame overhead at the end of processing. I'm going to try the above while waiting for your feedback. If that doesn't burn the house down, I'll try something more interesting.