How to use split inference on webcam via the Segmentation package?

[namas191297]

Hi there,

Thank you for the amazing repository, you're really doing some great work here.

I was going through the repository and have been playing around with the Segmentation task. I am particularly curious about the Segment method in SegmentationModel.cs file:

public IEnumerator Segment(Texture2D image, Action<Texture2D> return_callback)
        {
            var output_tensors = new Dictionary<string, Tensor>();
            yield return CoroutineHandler.StartStaticCoroutine(Predict(image, (outputs) => output_tensors = outputs));
            var output_name = runtime_model.outputs[0].name;
            var output_tensor = output_tensors[output_name] as TensorFloat;

            var indices_shape = new TensorShape(1, output_tensor.shape[2], output_tensor.shape[3]);
            var indices = TensorInt.AllocNoData(indices_shape);
            backend.ArgMax(output_tensor, indices, 1, false, false);

            output_tensor.CompleteOperationsAndDownload();
            indices.CompleteOperationsAndDownload();

            var indices_texture = ToTexture(indices);
            var resized_texture = Resize(indices_texture, image.width, image.height);

            indices.Dispose();
            output_tensors.AllDispose();
            MonoBehaviour.Destroy(indices_texture);

            return_callback(resized_texture);
        }

Would it possible for you to provide an example of how to call this function during Update() when running an application that uses Webcam frames for Inference?

Cheers, Namas

[UnaNancyOwen]

[!WARNING] The feature of split inference by multiple frames is still an experimental implementation.

[!WARNING] Segmentation has a known issue with misaligned results. This is still an open issue yet.

A simple example is here. Please specify how many layers to process per frame using model.SetLayersPerFrame().

private void Start()
{
    // Create Segmentation Model
    model = new HoloLab.DNN.Segmentation.SegmentationModel(weights);
    model.SetInputMean(mean);
    model.SetInputStd(std);
    model.SetLayersPerFrame(layers_per_frame); // specify process number of layers to process per frame (e.g. 50)

    /* Other Initializations */
}

private void Update()
{
    /* Get Texture2D from WebCamera */

    // Segment Area and Draw
    StartCoroutine(Segment(input_texture));
}

private IEnumerator Segment(Texture2D input_texture)
{
    // Segment Area
    Texture2D indices_texture = null;
    yield return StartCoroutine(model.Segment(input_texture, (output) => indices_texture = output));
    //var indices_texture = model.Segment(input_texture);

    // Draw Area on Unity UI
    var colorized_texture = HoloLab.DNN.Segmentation.Visualizer.ColorizeArea(indices_texture, colors);
    if (output_image.texture == null)
    {
        output_image.color = Color.white;
        output_image.texture = new Texture2D(indices_texture.width, indices_texture.height, TextureFormat.RGBA32, false);
    }
    Graphics.CopyTexture(colorized_texture, output_image.texture);

    // Destroy Texture
    Destroy(colorized_texture);
    Destroy(indices_texture);
}