pass through video Using Material From AR Camera BackGround (feat.cpuImage, graphics.blit)

[uvclyh]

Can I transfer passthrough video through material AR Camera background using graphics.blit method ?

I'm making Unity Client video chat application using AR Foundation, WebRTC A Difficulty is transfer camera Texture From Device to others I wrote a ARCamreader script component attached to AR Camera Game object

I did try two ways

1. Accessing the Camera Image on the CPU

it works, but few seconds after matching with counterpart(WEBRTC video) Apps would be crashed. I expecting memory leak during communication So that I try to use graphic.blit method from AR camera Background Script Component

2. graphic.blit with AR background component

Fortunatelry Okay Sending Video, Audio, Data, But Objects never Rendered on counterpart's Screen (meaning Counterpart can see No pass thourgh video, but just video Camera Rendered)

Device Screen(Android)

Counterpart's Screen

I think AR Camera Background's defualt material does not support passthough image, But editing custom Background Shader is not a good approach

Here is partial code

#if !UNITY_EDITOR 
    void OnEnable()
    {
        RenderPipelineManager.endCameraRendering += RenderPipelineManager_endCameraRendering;
    }
    void OnDisable()
    {
        RenderPipelineManager.endCameraRendering -= RenderPipelineManager_endCameraRendering;
    }

    private void RenderPipelineManager_endCameraRendering(ScriptableRenderContext context, Camera camera)
    {
        OnPostRender();
    }

    private void OnPostRender()
    {
        CopyARCameraTextureAndUpdateDeviceFrame(mUsedDeviceName);
    }
#endif

    private void CopyARCameraTextureAndUpdateDeviceFrame(string DeviceName)
    {
        Graphics.Blit(null, targetRenderTexture, m_ARCameraBackground.material);

        var activeRenderTexture = RenderTexture.active;
        RenderTexture.active = targetRenderTexture;
        if (m_LastCameraTexture == null) // for WEBRTC TextureFomat should be RGBA32
            m_LastCameraTexture = new Texture2D(targetRenderTexture.width, targetRenderTexture.height, TextureFormat.RGBA32, true); 
        m_LastCameraTexture.ReadPixels(new Rect(0, 0, Screen.width, Screen.height), 0, 0);
        m_LastCameraTexture.Apply();
        RenderTexture.active = activeRenderTexture;
        ARTexture.texture = m_LastCameraTexture; // RawImage for Debugging
        //  for WEBRTC transmission video
        mVideoInput.UpdateFrame(DeviceName, m_LastCameraTexture.GetRawTextureData(), m_LastCameraTexture.width, m_LastCameraTexture.height, WebRtcCSharp.ImageFormat.kABGR, 0, true);
    }

Unity Version : 2021.1.17f1 ARFoundation : 4.1.9

[DavidMohrhardt]

Just to clarify what you are trying to do is capture the final render correct? Then you don't need to use the ARCameraBackground material to perform the blit but instead simply take your camera's target RenderTexture as the source texture of Graphics.Blit(Texture src, Texture dest);.

The ARCameraBackground material is pretty strictly just for rendering the camera background texture before opaques get rendered. It's not setup to handle rendering over augmented content.

[uvclyh]

@DavidMohrhardt Thanks for advice! According to your advice, I took AR camera render texture as src texture, and it works

What i want is to streaming camera texture to others through webRTC. I thought src texture cannot be a render texure in Graphics.blit method

Thanks~

Sharing edited code

[ZFHung]

this is my Code ，App open Black Screen，how to fix it?

void AR_Foundation_Device_SetUp() { mediaStream = new MediaStream(); videoStreamTrack = ARCamera.CaptureStreamTrack(1280, 720); mediaStream.AddTrack(videoStreamTrack); }