DLL Import Exception in Unity Editor

[task-master98]

Hi, So I am currently using the Unity example project, which is helping my research immensely! I just have 1 doubt: so when I click play on the example unity project, the debug logs give out a DLLNotFoundException for the hl2da dll file. This is obviously included in the plugins folder. However, when I deploy the same to the Hololens, it works. How is it possible? Can you explain the discrepancy with the Unity editor and the Hololens device? Thanks!

[task-master98]

Also another silly doubt! All the feeds including the RGB webcam appear to be red in color: is this expected? Can I change the the color to be grayscale instead for the VLC and normal RGB for the webcam? Also I am assuming that the RGB camera is called PV in the code. Is this correct?

[jdibenes]

Hello, I do not know, sorry. I assume Unity is trying to run the project locally as x64 and attempting to load the x64 version of the hl2da plugin, which does not exist. For quick testing, all images are copied into single channel R8/R16 (red) textures. Not sure if there is a grayscale texture format, have not looked into it. If there is none, then it may have to be converted to RGB using a shader that copies the red channel to the green and blue channels. PV is the color camera as you said. For PV, the image is returned as NV12 so it has to be converted to RGB and copied into a RGB texture but this is not implemented yet.

[task-master98]

Hi, Thanks a lot for the response. I actually tried implementing the shader myself. Now the video seems to be oriented correctly (it is no longer flipped). However, I cannot seem to convert the YUV to RGB for the PV camera. The image appears to be mostly pink and yellow. Here is the code for the shader file and how I am using it.

Shader "Custom/NV12toRGB"
{
    Properties
    {
        _MainTex ("Y Plane", 2D) = "white" {}
        _UVTex ("UV Plane", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            // make fog work
            sampler2D _MainTex;
            sampler2D _UVTex;

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float2 uv : TEXCOORD0;                
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;
                return o;
            }

            half4 frag (v2f i) : SV_Target
            {
                float2 uv = float2(i.uv.x, 1.0 - i.uv.y);

                float Y = tex2D(_MainTex, uv).r;
                float U = tex2D(_UVTex, uv).r - 0.5;
                float V = tex2D(_UVTex, uv).a - 0.5;

                float3 rgb;
                rgb.r = Y + 1.3983 * V;
                rgb.g = Y - 0.39465 * U - 0.58060 * V;
                rgb.b = Y + 2.03211 * U;

                return half4(rgb, 1.0);
            }
            ENDCG
        }
    }
    FallBack "Diffuse"
}

Here is the method to update PV:

 void UpdatePV(hl2da_framebuffer fb)
    {
        int width = pvcf.width;
        int height = pvcf.height;
        int ysize = width * height;
        int uvsize = ysize / 2;

        tex_pv.LoadRawTextureData(fb.Buffer(0), ysize * sizeof(byte)); // Image is NV12, load Y only, use OpenCV to convert NV12->RGB
        tex_pv.Apply();

        IntPtr uvptr = fb.Buffer(0) + ysize * sizeof(byte);
        Debug.Log("UV-plane pointer computed");
        tex_uv.LoadRawTextureData(uvptr, uvsize * sizeof(byte));
        tex_uv.Apply();
        // Display frame
        Material mat = pv_image.GetComponent<Renderer>().material;
        mat.SetTexture("_MainTex", tex_pv);
        mat.SetTexture("_UVTex", tex_uv);
        Debug.Log("Texture set");
    }

Thanks a lot!

[jdibenes]

Hello, The plugin now includes a function for NV12 to RGB conversion. The code seems OK, what is the format of the tex_uv texture?

[task-master98]

It is simply a Texture2D here is the declaration:

// declaration
private Texture2D tex_uv;

// In void Start method

tex_uv       = new Texture2D(pvcf.width / 2, pvcf.height / 2, TextureFormat.RG16, false);

I see that you have recently added a commit for the YUV to RGB conversion. Can you provide an example of how I may use it?

[jdibenes]

Sure, here is an example https://github.com/jdibenes/hl2da/blob/6b2317c24cb57c7a2441aea481b91ecb5b07681b/hl2da_unity/Assets/Scripts/HoloLens2DA.cs#L491 So if the texture format is RG16 shouldn't V be tex2D().g in the shader?

[task-master98]

Thanks a lot! That seems to have fixed the issue!

[task-master98]

Hi I am experimenting with this repo. I am updating the data from PV on a quad. I am trying to scale up the size of the quad. However, when I do that the video from PV becomes very glitchy and does not update in real time. When the quad had a smaller size, I did not face such a problem. Is this a resolution issue? How can I resolve this?

[jdibenes]

Hello, What PV resolution are you using? Resolutions with a width that is not a multiple of 64 require that the destination memory (e.g., texture) is padded so the width is a multiple of 64. For example, for PV resolution 760x428 the texture size should be 768x428. hl2da_imt.GetStride_PV computes this value from width.