Windows Mobile - Memory Leaks on Nokia Lumia Phones

[paulmorrishill]

Tested on the Nokia Lumia 625 There are several significant memory leaks in the windows universal app implementation of the plugin. Specifically around the MediaCapture API and it's uses.

They have been identified by others see here and here and I have personally verified them.

The solution of clearing off the audio did not work for me.

The problem revolves around any attempt to take an image from the camera. I have created a fixed version of the plugin but it involved rewriting a substantial portion of the code and changing the mechanism by which it takes images. I would submit a pull request but I don't have another phone to test and I don't own the IP rights to this code.

See here for the doc on MediaCapture

There are serious leaks in Insantiating a MediaCapture object The CapturePhotoToStorageFileAsync method The CapturePhotoToStreamAsync method

There appear to be no leaks (or very small leaks) in the API for obtaining the preview stream.

The solution This is how I worked around these memory leaks. First of all Instantiate a MediaCapture object on the first call and reuse that same object for all subsequent requests.

Currently the windows implementation feeds the preview stream into a video element, while in the background takes pictures every couple of seconds and analyses them for barcodes, this also has the annoying shutter sound too.

Instead of this, feed the stream into the video element then use canvas.drawImage to copy an image from the video element into an html canvas element. Then call getImageData() on the canvas and feed the byte array straight into the ZXing library.

Note that when using drawImage you MUST specify width and height otherwise you will get a black image - contrary to the HTML specification for this method.

[eisenbergrobin]

I'd like to submit a PR to implement this, as I need leak-less barcode scanning on a Lumia 1320 for a production application. I have a few questions:

Isn't it a little convoluted to drawImage from the Javascript side? That means the whole data flow is moving back and forth between the javascript side and the native side, and that seems costly. Since we are going to be feeding the image back to the native side in any case to use Zxing, why not take the image data directly on the native side? Is there not a functional mechanism on Lumia phones to get a byte array from a camera preview without triggering the shutter and all?

Could you post (or send me a patch to) your PR maybe? I have a few 1320s to test this on.

[paulmorrishill]

The data is currently being fed into the video element from the native side at maximum resolution purely to display the preview, so the overhead between JS and native is almost non existent. From my testing I didn't see any significant lag at all so I wouldn't be concerned about going back and forth between JS and native. There may be a way to do it native side but I couldn't see a way at the time. Also bear in mind that the JS API is supposed to be the same as using the CLR API in universal apps, unlike with normal cordova running in a web view.

I can't send you the code unfortunately as my company has a strict policy on code leaving the company.

With the information above it shouldn't be hard to replicate though. Only about 20-30 lines. Located in src\Windows\BarcodeScannerProxy.js src\Windows\Lib\Reader.cs

[eisenbergrobin]

Yeah I should be able to reproduce it pretty quickly. Thanks for your help!

[tony--]

@eisenbergrobin - did you make any progress on replicating the approach that @paulmorrishill described?

[eisenbergrobin]

Yes I did. I have a working scanner without leaks on UWP, but the performance is not good.

I only have a Lumia 1320 to test, and basically it does not autofocus correctly, so I have to trigger autofocus manually and scan for a barcode on focus completion, which means the scan is jittery.

Also, the performance of ZXing when used in this manner is not acceptable: I'm not getting results on clear images that should be giving results.

According to my tests, the memory leaks in the Capture API aren't that pronounced, if you correctly pause and close the capture correctly, my camera doesn't reach the 'inoperable' green-screen state that it does on the current master branch of the plugin.

All in all I'm not sure I'll be able to get a satisfying scan from @paulmorrishill's approach: IIRC ZXing has detection methods that scan the preview stream in real-time and algorithmically choose when to decode the image. Here we are asking for decoding every time, there is no smart detection of barcode presence or position.

I'm not ready to submit a PR on this, but @tony-- you can message me if you need help getting a working scanner under WP, I have one here in under 150 lines of code for the whole plugin.

If anybody has any better ideas for getting a working, performant scanner under UWP I'm all ears.

[sgo13]

Hi @eisenbergrobin,

I have some problems to get a working scanner under Windows Phone 8.1. There are memory leaks on Nokia Lumia 625 / 925, and I'd like to implement the solution given by @paulmorrishill to change the mechanism by which the scanner takes images.

Can you help me please ?

[eisenbergrobin]

Ok here's a rundown of what I'm doing, to help you until I can find a suitable solution and submit a PR.

Reader.cs is extremely simple:

All it does is take a bytearray representing an image from the camera and try to decode a barcode.

namespace WinRTBarcodeReader
{
    using System;
    using System.IO;
    using System.Runtime.InteropServices.WindowsRuntime;
    using System.Threading;
    using System.Threading.Tasks;
    using Windows.Foundation;
    using Windows.Graphics.Imaging;
    using Windows.Media.Capture;
    using Windows.Media.MediaProperties;
    using Windows.Storage.Streams;
    using Windows.UI.Xaml.Media.Imaging;
    using ZXing;

    /// <summary>
    /// Defines the Reader type, that perform barcode search asynchronously.
    /// </summary>
    public sealed class Reader
    {
        #region Private fields

        /// <summary>
        ///     Data reader, used to create bitmap array.
        /// </summary>
        private BarcodeReader _barcodeReader;

        #endregion

        #region Constructor
        public Reader()
        {
            _barcodeReader = new BarcodeReader { AutoRotate = true, TryInverted = false, Options = { PureBarcode = false, TryHarder = true, PossibleFormats = new BarcodeFormat[] { BarcodeFormat.EAN_13, BarcodeFormat.QR_CODE, BarcodeFormat.CODE_128 } } }; // Change formats according to what you need. 
        }

        #endregion

        #region Public methods
        public Result decodeRaw([ReadOnlyArray] byte[] imageData)
        {            
            Result _result = _barcodeReader.Decode(imageData, 720, 1230, BitmapFormat.RGBA32);
            return _result;
        }

        #endregion
    }
}

Then the JS side makes a canvas, attaches the capture, and takes snapshots from the capture, to send them back to Reader.cs for decoding:

This function initializes a camera on a target canvas passed as a parameter:

 function setupCapture(target) {
        window._wpCameraCapture = new Windows.Media.Capture.MediaCapture();
        var deviceInfo = Windows.Devices.Enumeration.DeviceInformation;
        var captureSettings = new Windows.Media.Capture.MediaCaptureInitializationSettings();
        captureSettings.streamingCaptureMode = Windows.Media.Capture.StreamingCaptureMode.video;
        captureSettings.photoCaptureSource = Windows.Media.Capture.PhotoCaptureSource.videoPreview;

        //This allows for switching between different cameras. For my Lumia 1320, the back camera is n°1, the front camera is n°0
        return deviceInfo.findAllAsync(Windows.Devices.Enumeration.DeviceClass.videoCapture).then(function (devices) {
            captureSettings.videoDeviceId = devices[1].id;
            captureSettings.audioDeviceId = "";
            return window._wpCameraCapture.initializeAsync(captureSettings).then(function () {
                var controller = window._wpCameraCapture.videoDeviceController;
                controller.torchControl.enabled = false;
                controller.flashControl.enabled = false;
                controller.flashControl.auto = false;

                if (controller.focusControl && controller.focusControl.supported) {
                    if (controller.focusControl.configure) {
                        var focusConfig = new Windows.Media.Devices.FocusSettings();
                        focusConfig.autoFocusRange = Windows.Media.Devices.AutoFocusRange.macro;
                        var supportContinuousFocus = controller.focusControl.supportedFocusModes.indexOf(Windows.Media.Devices.FocusMode.continuous).returnValue;
                        var supportAutoFocus = controller.focusControl.supportedFocusModes.indexOf(Windows.Media.Devices.FocusMode.auto).returnValue;

                        if (supportContinuousFocus) {
                            focusConfig.mode = Windows.Media.Devices.FocusMode.continuous;
                        } else if (supportAutoFocus) {
                            focusConfig.mode = Windows.Media.Devices.FocusMode.auto;
                        }

                        controller.focusControl.configure(focusConfig);
                    }
                }

                //Capture has been correctly initialized. Start HTML Canvas preview:
                var context2d = target.getContext("2d");

                window._wpCameraCapturePreview = document.createElement("video");
                window._wpCameraCapturePreview.style.cssText = "width: 720px; height: 1230px;";
                window._wpCameraCapturePreview.src = URL.createObjectURL(window._wpCameraCapture);
                window._wpCameraCapturePreview.play();

                function animationFrame() {
                    if (target && window._wpCameraCapturePreview) {
                        context2d.translate(720, 0);
                        context2d.rotate(Math.PI / 2);
                        context2d.drawImage(window._wpCameraCapturePreview, 0, 0, 1230, 720);
                        context2d.setTransform(1, 0, 0, 1, 0, 0);
                        window.requestAnimationFrame(animationFrame);
                    }
                }
                window.requestAnimationFrame(animationFrame);
            });
        });
    }

A few notes on this:

• I'm finding the videoDeviceId statically, as my target is a single phone (Lumia 1320). There is probably something that you need to do to select the right camera on your devices
• All sizes are static for the same reason, there is however no reason not to do something that scales according to the target canvas.
• I can't get a correct camera rotation to work, so I rotate the HTML canvas manually. If anybody has a better idea I'm all ears.

Now for the actual decoding: I can't get autofocus to work on the Lumia 1320 I have. That means I have no reliable way to discern the right moment for triggering a ZXing decode operation. So what I've chosen to do is to trigger the camera focus recursively, and decode on focus completion. I'm sure something better can be done using the AutoFocusCompleted event on devices that receive it. I never receive this event on the Lumia 1320.

function scanImageForBarcode(rawImage) {
        var barcodeReader = new WinRTBarcodeReader.Reader();
        var result = barcodeReader.decodeRaw(rawImage.data);
        return result;
    }

function continuouslyDecodeCameraPreview(targetContext, success) {
        var triggerFocus = function () {
            try {
                window._wpCameraCapture.videoDeviceController.focusControl.focusAsync().done(function () {
                    var imageBytes = targetContext.getImageData(0, 0, 720, 1230);
                    var result = scanImageForBarcode(imageBytes);
                    if (result) {
                        success(result);
                    }
                    triggerFocus();
                });
            } catch (e) {
                console.log("Caught ", e)
            }
        };
        triggerFocus();
}

The code above triggers focus recursively, and tries to decode the image upon focus completion. If ZXing decodes a barcode, the result is passed to the success callback.

The whole system is called like this:

 setupCapture(target).then(function () {
                setTimeout(continuouslyDecodeCameraPreview.bind(this, target.getContext("2d"), success), 500);
            });

Now for the important part:

The memory leak that is talked about in this thread is in the wpCameraCapture object. If the capture is not closed explicitely, it leaks. But if you free these resources correctly there is no discernable effect from the leak. Which is why all my variables are on window, so I can clean them up effectively at all times:

function cleanUp() {
        window._wpCameraCapturePreview.pause();
        window._wpCameraCapturePreview.src = null;
        window._wpCameraCapturePreview = null;
        window._wpCameraCapture.close();
        window._wpCameraCapture = null;
        console.log("Cleaned up!");
};

And I set this cleanup to happen in all cases of end of plugin aciton (sucess, failure) as well as setting it on window.onclose. As long as cleanUp() is called I don't get any leaks. If I forget to free these resources even once, the camera is unusable on the phone until reboot (green screen).

That should give you all you need to implement a scan of your own.

Caveats:

• The camera focuses and refocuses all the time: this makes for sub-par performance.
• ZXing has to rotate the image to decode, making it slower. I have not been able to initiate a correct portrait capture to get the correct orientation from the start.
• If the application crashes during a scan, capture resources are not freed and the camera will become unusable.

I'm open to any and all suggestions on how to fix these problems. I'll be submitting a clean PR when these problems are fixed for all WP devices.

Hope I helped :)

[tony--]

@eisenbergrobin - Thanks for the detailed write up!

[sgo13]

Thank you for your help @eisenbergrobin , I'm going to try this.

[paulmorrishill]

Hi @eisenbergrobin, I've got some snippets you might find useful

Find media device ID

                //4 = media devices
                Windows.Devices.Enumeration.DeviceInformation.findAllAsync(4).then(function (devices) {
                    for (var i = 0; i < devices.length; i++) {
                        var dev = devices[i];
                        if (dev.enclosureLocation == null) continue;
                        if (dev.enclosureLocation.panel == null) continue;
                        if (dev.enclosureLocation.panel !== Windows.Devices.Enumeration.Panel.back) continue;
                        captureSettings.videoDeviceId = dev.id;
                    }
                    capture.initializeAsync(captureSettings).done(afterCaptureInit);
                }, function () { });

Set rotation

            controller.setMediaStreamPropertiesAsync(Windows.Media.Capture.MediaStreamType.videoRecord, maxResProps).done(function () {
                    // handle portrait orientation
                    if (Windows.Graphics.Display.DisplayProperties.nativeOrientation == Windows.Graphics.Display.DisplayOrientations.portrait) {
                        capture.setPreviewRotation(Windows.Media.Capture.VideoRotation.clockwise90Degrees);
                    }

                });

Also I don't agree that closing the objects down and disposing them releases all resources as a tried this, the memory leak is much smaller but still exists. I set up a test app that continuously scanned and monitored the memory usage, it will always increase every time you instantiate the MediaCapture object. I even wrote a test that literally just instantiated this a MediaCapture object disposed it and repeated, memory usage increased continuously.

[eisenbergrobin]

Hi @paulmorrishill, thanks for these.

The first snippet should work for everyone, for finding the correct camera on your WP device.

The second I tried, and even if I set the mediaProperties to use the correct resolution I end up with a preview stream that is not in the correct aspect ratio. On my Lumia 1320, calling capture.setPreviewRotation to 90deg makes the preview stream take the wrong size. If someone has a working implementation of preview rotation (at least on 1320) I'm interested.

As for the leak: yes, I still leaks. It just leaks less. IIRC since the application memory is sandboxed though, this memory should be marked as freed when the application is closed. For my needs, I need to be able to scan at most about 20 scans per hour for 8 hours per day (I'm building an application for retail stores). If I correctly close these resources, I can manage that many scans, and it is enough for the needs of my client. But I can understand how that might not be enough for some, and I'm not surprised that programmatic testing shows a definite and measurable leak: those who prefer it can choose to instanciate the capture only once.

On another note, the 'green screen' that I get if I don't close the capture is a serious bug: the camera becomes unusable until the device has rebooted.

[sgo13]

Hi,

When I use capture.setPreviewRotation to 90deg, the camera preview is well oriented, but when I use canvas.drawImage to copy image, the scale is not preserved.

Do you know this problem ? Do you know how to fix it ?

[eisenbergrobin]

Yes this is the rotation problem I was talking about.

I have not been able to get a correct working portrait scan using this method. When I call setPreviewRotation (tried all variations, 90 & 270) I get a preview frame that is distorted. That is why I ask the ZXing Decoder to AutoRotate the image.

@paulmorrishill maybe you've got an idea?

[sgo13]

Do you know if there is a workaround or another method / plugin to read barcode on Windows phone ? My project is on standby because of that problem, so I have to find any way to read barcode to unblock the situation ...

[eisenbergrobin]

@sgo13 I have not found a workaround or another plugin.

Just a few questions for anyone who has a WP8.1U1 device other than the Lumia 1320 (that is all I have):

• Does autofocus or continuous focus work?
  • If so, is the FocusChanged event ever fired ? Does videoDeviceController.focusControl.focusState ever change?
• Does camera rotation, as shown by @paulmorrishill work for you? For me it messes up the capture aspect ratio (with a 90 or 270 degree rotation).
• Does your phone support camera captures without triggering flash and focus?

I'm still commited to submitting a PR when I have a correct working implementation of barcode scanning on WP, but for now I don't consider my implementation to be correct. It works, but performance is abysmal when compared to iOS or Android devices (even low-range devices).

I have a meeting with a developer liaison from Microsoft tomorrow to try to get these problems sorted out. I'll keep you updated if we achieve something correct.

[paulmorrishill]

I've tried on the 625 and the 735 and the rotation works Ok for me in portrait mode, landscape mode is rotated incorrectly but for my application I haven't needed to look into it.

Are you using mszoom?

capturePreview.addEventListener("playing", function() {
                capturePreview.msZoom = false;
                capturePreview.msZoom = true;
            });

Without this it doesn't scale fully to the video element for me. Haven't had a problem with focusing and I'm using the same focus code as you which I think was part of the original plugin.

if (controller.focusControl && controller.focusControl.supported) {
                if (controller.focusControl.configure) {
                    var focusConfig = new Windows.Media.Devices.FocusSettings();
                    focusConfig.autoFocusRange = Windows.Media.Devices.AutoFocusRange.macro;

                    var supportContinuousFocus = controller.focusControl.supportedFocusModes.indexOf(Windows.Media.Devices.FocusMode.continuous).returnValue;
                    var supportAutoFocus = controller.focusControl.supportedFocusModes.indexOf(Windows.Media.Devices.FocusMode.auto).returnValue;

                    if (supportContinuousFocus) {
                        focusConfig.mode = Windows.Media.Devices.FocusMode.continuous;
                    } else if (supportAutoFocus) {
                        focusConfig.mode = Windows.Media.Devices.FocusMode.auto;
                    }

                    controller.focusControl.configure(focusConfig);
                    controller.focusControl.focusAsync();
                }
            }

No flash is triggered either on the two phones that I've tested it on.

[daserge]

This is still an issue with some Windows Phone 8.1 upd.1 devices (Nokia Lumia 525). Continuous scanning results in the error:

Scanning failed: WinRTError: Not enough storage is available to complete this operation.

System.OutOfMemoryException: Insufficient memory to continue the execution of the program.
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at WinRTBarcodeReader.Reader.<GetCameraImage>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerS

Subsequent scan attempt results in the app and camera driver crash (green screen).

[abhijithp]

@eisenbergrobin i have some problem barcode scanning so can you give a jsfiddle example for the above method.