Landscape webcam texture to portrait screen

[DutchRose-Jochem]

Hi,

First, thanks for this great sample project!

I am experiencing an issue though. I'm trying to show the webcam texture on a portrait display. Currently I'm working on an iMac.

I've edited the WebcamInput script to change the horizontal scale based on the aspect ratio of the screen, so the webcam texture doesn't get squished. This works, but when I change the display to a 1080x1920 portrait screen the quality of the segmentation drastically deteriorates. While a square display (1920x1920) works just fine.

Do you have any idea what the issue might be?

[keijiro]

My guess is that your input image is wrongly rotated. I'd recommend checking if you're giving a correctly rotated image to the NN model.

[keijiro]

I'm closing this issue now. Please reopen it for further problems.

[DutchRose-Jochem]

Hm, OK, but I'm not rotating anything. So how can I check if the rotation of the image is correct?

[keijiro]

You're using portrait mode, so I assumed that the input was 90° rotated.

So how can I check if the rotation of the image is correct?

I'm not sure because I don't know how you implemented the application.

[DutchRose-Jochem]

I get the issue using your sample and only changing the webcaminput script. See below. Then in Play Mode I change the display resolution to 1080x1920. If it's easier I can also send you the whole project, just let me know.

How I calculate the offset isn't really correct at the moment, I'm also trying to figure that one out at the moment.

using UnityEngine;

namespace NNCam
{

    public sealed class WebcamInput : MonoBehaviour
    {
        #region Editable attributes

        [SerializeField] string _deviceName = "";

        #endregion

        #region Internal objects

        WebCamTexture _webcam;
        RenderTexture _buffer;

        #endregion

        #region Public properties

        public Texture Texture => _buffer;

        #endregion

        #region MonoBehaviour implementation

        float webcamWidth = 1920;
        float webcamHeight = 1080;
        float horizontalScale = 1;

        void Start()
        {
            //_webcam = new WebCamTexture(_deviceName);
            _webcam = new WebCamTexture((int)webcamWidth, (int)webcamHeight, 30);
            _buffer = new RenderTexture((int)webcamWidth, (int)webcamHeight, 0);
            _webcam.Play();
        }

        void OnDestroy()
        {
            Destroy(_webcam);
            Destroy(_buffer);
        }

        void Update()
        {
            horizontalScale = (webcamHeight / webcamWidth) * ((float)GetComponent<Camera>().pixelWidth / (float)GetComponent<Camera>().pixelHeight);

            if (!_webcam.didUpdateThisFrame) return;
            var vflip = _webcam.videoVerticallyMirrored;
            var scale = new Vector2(-horizontalScale, vflip ? -1 : 1);
            var offset = new Vector2(horizontalScale * 1.5f, vflip ? 1 : 0);
            Graphics.Blit(_webcam, _buffer, scale, offset);
        }

        #endregion
    }

} // namespace NNCam

[keijiro]

You shouldn't change the aspect ratio of the input texture. It reduces the accuracy of the prediction.

[DutchRose-Jochem]

Oh, OK. Then do you have any idea what would be the best way to make it work on a portrait display?

[keijiro]

Padding (adding black empty spaces to the both side of the image) is mostly used. I think you should look for other better models with portrait support, though.

[DutchRose-Jochem]

I will look into that, thanks.