Kas-code
commented
3 years ago I'm using the following method to try to detect the timeout. It runs a Task.Delay as well as cam.ProcessAsync at the same time, and then uses Task.WhenAny to wait for either of them to finish. If the Task.Delay returns first then it knows that cam.ProcessAsync is hanging.
public static async Task<bool> CancelAfterAsync(Task startTask, TimeSpan timeout)
{
using (var timeoutCancellation = new CancellationTokenSource())
{
var delayTask = Task.Delay(timeout, timeoutCancellation.Token);
Serilog.Log.Logger.Debug("await Task.WhenAny");
var completedTask = await Task.WhenAny(startTask, delayTask);
Serilog.Log.Logger.Debug("Finished await Task.WhenAny");
// Cancel timeout to stop either task:
// - Either the original task completed, so we need to cancel the delay task.
// - Or the timeout expired, so we need to cancel the original task.
// Canceling will not affect a task, that is already completed.
timeoutCancellation.Cancel();
if (completedTask == startTask)
{
// original task completed
Serilog.Log.Logger.Debug(" await startTask;");
await startTask;
return true;
}
else
{
Serilog.Log.Logger.Debug("Timed out");
// timeout
return false;
}
}
}I then use this in the following way:
public static async Task<byte[]> CaptureRawData()
{
MMALCamera cam = MMALCamera.Instance;
MMALCameraConfig.StillEncoding = MMALEncoding.BGR24;
MMALCameraConfig.StillSubFormat = MMALEncoding.BGR24;
using (var imgCaptureHandler = new MemoryStreamCaptureHandler())
using (var renderer = new MMALNullSinkComponent())
{
cam.ConfigureCameraSettings(imgCaptureHandler);
cam.Camera.PreviewPort.ConnectTo(renderer);
// Camera warm up time
await Task.Delay(2000);
Serilog.Log.Logger.Debug("cam.ProcessAsync(cam.Camera.StillPort)");
if (await TaskUtils.CancelAfterAsync(cam.ProcessAsync(cam.Camera.StillPort), TimeSpan.FromSeconds(100)))
{
Serilog.Log.Logger.Debug("Got response, returning data");
return imgCaptureHandler.CurrentStream.ToArray();
}
else
{
Serilog.Log.Logger.Warning("Camera timed out, return null");
return null;
}
}
}
public static async Task<byte[]> CaptureImage()
{
byte[] data = null;
for (var i = 0; i < 10; i++)
{
data = await CaptureRawData();
if (data != null)
{
break;
}
else
{
Serilog.Log.Logger.Warning("Camera timed out, retrying");
}
}
if (data == null || data.Length == 0)
{
//Todo: Better exception message
throw new Exception("Image capture failed");
}
return data;
}If the cam.ProcessAsync method hangs, then the CancelAfterAsync method should detect this and return false. When this returns false, the CaptureRawData method should return null. Then the CaptureImage method should detect a null result and retry up to 10 times. But instead, the logging output I get is the following:
[13:54:54 DBG] Running CaptureRawData [13:54:54 DBG] cam.ProcessAsync(cam.Camera.StillPort) [13:54:54 DBG] await Task.WhenAny [13:56:34 DBG] Finished await Task.WhenAny [13:56:34 DBG] Timed out [13:56:34 WRN] Camera timed out, return null
It then hangs indefinitely on the return null line inside CaptureRawData. This makes no sense. It's clear that CancelAfterAsync has detected the hang and returned false, but then it hangs when returning null from CaptureRawData. It never gets to the line where it logs "Camera timed out, retrying", which is confusing because all it's doing is returning null?
If anyone can just help me with how to detect the hang and retry safely then that would be great.
techyian
Hello, I'm running the following code on my Raspberry Pi:
This captures an image to memory and returns it as a byte array. Mostly it works great. I'm running this to capture an image every 10 seconds or so. Randomly, after a few hours the program freezes on the line "await cam.ProcessAsync" which I can tell because of the serilog logging. Having left it for several hours, it never recovers, it just hangs forever on that line and I have to Ctrl+C to kill the program. If I run it again it can capture images again without issue, only to hang again after a few hours in the same place. I read that this may be caused due to power issues, but I'm using an official RPi power supply without any other peripherals. Regardless, even if it is caused by intermittent power brown out, it doesn't really matter to me if it fails to capture one image every few hours, I just want to be able to detect the failure and try again. As it stands, I can't retry since the main thread completely locks up. It's an absolute requirement that I capture the image to memory, I can't use the other methods in MMALSharp to capture the image to disk, since I'm doing some post filtering on the image in memory after capturing it, and then deciding that I only want to save certain images to disk if they pass certain tests. I've attempted to detect the hang with a timeout and retry, but this also hangs with some strange behaviour. This is already quite a long post so I'll post as a reply my attempt to work around the issue.