Unity Barracuda EXC_RESOURCE RESOURCE_TYPE_MEMORY (limit=2098 MB, unused=0x0)

[marcomameli1992]

Hi I'm trying to use the SSD network with mobilenetv2 (with ReLU instead of ReLU6 as activation function) to insert object detection in my AR Application but when I try to run the application on my iPhone XS Max I receive the error in the title. I do not understand why and what I'm doing wrong. Here I report my code:

using System;
using System.Collections;
using System.Collections.Generic;
using Unity.Collections.LowLevel.Unsafe;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.InputSystem;

public class ARFrameManager : MonoBehaviour
{
    // On frame event
    public delegate void FrameObtained(Texture2D image);
    public static event FrameObtained OnFrameObtained;

    // ArCamera manager is used to acces each frame from the smartphone's camera
    private ARCameraManager _arCameraManager;

    private void Awake()
    {
        _arCameraManager = gameObject.GetComponentInChildren<ARCameraManager>();
    }

    private void OnEnable()
    {
        _arCameraManager.frameReceived += OnReceivedFrame;
        GameManager.StopFrameEvent += PauseFrameAcquisition;
        GameManager.StartFrameEvent += StartFrameAcquisition;
    }

    private void OnDisable()
    {
        _arCameraManager.frameReceived -= OnReceivedFrame;
        GameManager.StopFrameEvent -= PauseFrameAcquisition;
        GameManager.StartFrameEvent -= StartFrameAcquisition;
    }

    void PauseFrameAcquisition()
    {
        _arCameraManager.frameReceived -= OnReceivedFrame;

    }

    void StartFrameAcquisition()
    {
        _arCameraManager.frameReceived += OnReceivedFrame;
    }

    unsafe void OnReceivedFrame(ARCameraFrameEventArgs arCameraFrameEventArgs)
    {
        /*     Attempt to get the latest camera image. If this method succeds
            it acquires a native resource that must be disposed
         */

        Texture2D image = arCameraFrameEventArgs.textures[0];

        //Debug.Log("ARFrameManager -> The Texture2D obtained has a shepe: (" + image.width + "," + image.height + ")");

        OnFrameObtained?.Invoke(image);
    }
}

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;

public class GameManager : MonoBehaviour
{
    // Object detection request event
    public delegate void ObjectDetectionInference(Texture2D iamge);
    public static event ObjectDetectionInference OnFrameCaptured;

    // On frame received event
    public delegate void FrameReceived();
    public static event FrameReceived OnFrameReceived;

    // On object detection return event
    public delegate void OnBBInferenceReturned();
    public static event OnBBInferenceReturned OnBoundingBoxReturned;

    // Frame acquisition stop event
    public delegate void StopFrameAcquisition();
    public static event StopFrameAcquisition StopFrameEvent;

    // Frame acquisiion restart event
    public delegate void StartFrameAcquisition();
    public static event StartFrameAcquisition StartFrameEvent;

    // 

    private void OnEnable()
    {
        ARFrameManager.OnFrameObtained += OnFrameReceive;
        ModelManager.OnBBoxObtained += OnBBInferenced;
    }

    private void OnFrameReceive(Texture2D image)
    {
        StopFrameEvent?.Invoke();
        OnFrameCaptured?.Invoke(image);
    }

    private void OnBBInferenced()
    {
        StartFrameEvent?.Invoke();
    }
}

using System;
using System.Collections;
using System.Collections.Generic;
using Unity.Barracuda;
using UnityEngine;

public class ModelManager : MonoBehaviour
{
    [Tooltip("The model asset path")]
    public NNModel modelAsset;

    private Model ssd300Model;
    private IWorker _worker;

    public delegate void OnBBoxInferenced();
    public static event OnBBoxInferenced OnBBoxObtained;

    private void OnEnable()
    {
        GameManager.OnFrameCaptured += GetBBox;
    }

    private void OnDisable()
    {
        GameManager.OnFrameCaptured -= GetBBox;
    }

    // Start is called before the first frame update
    void Start()
    {
        ssd300Model = ModelLoader.Load(modelAsset);
        _worker = WorkerFactory.CreateWorker(WorkerFactory.Type.ComputePrecompiled, ssd300Model);
    }

    // Update is called once per frame
    void Update()
    {

    }

    void GetBBox(Texture2D image)
    {
        if (image is null)
        {
            Debug.Log("ModelManager -> Image is null");
        }
        image.Resize(416, 416);
        image.Apply();
        Tensor inputImageTensor = new Tensor(image,3);
        _worker.Execute(inputImageTensor);
        Tensor output = _worker.PeekOutput();

        //Debug.Log("ModelManager -> TensorOutPut obtained: " + output.name);
        OnBBoxObtained?.Invoke();

    }
}

my script are used as component for Ar Session origin with the use of the model uploaded here: SSDLite

[mantasp]

It just means that application ran into hard Metal memory allocation limit on that device. There could be several reasons for that: leaking of GPU resources or just networking being too big for the device. Typically I would recommend to first try your model on desktop Mac and check how much memory does it use. You can obtain basic memory usage stats by calling _worker.Summary().

[marcomameli1992]

Ok thank you. To make this experiment I need to do not use the AR Foundation or it can work with macOS? I would like to have an opinion about the code. and some suggestion to manipulate the output of the network. Where I can find some examples?

[AlexRibard]

I would refer to our docs: https://github.com/Unity-Technologies/barracuda-release/blob/release/1.0.0/Documentation%7E/ModelOutput.md

Your code looks correct. You should be able to use the same model loading/execution on MAC. Although not an official tutorial, this project should give you a rough skeleton: https://github.com/AlexRibard/Barracuda-U-2-NetTest

[FlorentGuinier]

@marcomameli1992 Did the above solve your problem? :)

[amirebrahimi]

Closing since some suggestions were given, but there is no more activity on the thread, please reopen if needed.