Multhithreading D3D11 Corruption

[Taghazog]

Error output:

D3D11 WARNING: ID3D11DeviceContext::OMSetRenderTargets: Resource being set to OM RenderTarget slot 0 is still bound on input! [ STATE_SETTING WARNING #9: DEVICE_OMSETRENDERTARGETS_HAZARD] D3D11 WARNING: ID3D11DeviceContext::OMSetRenderTargets[AndUnorderedAccessViews]: Forcing PS shader resource slot 1 to NULL. [ STATE_SETTING WARNING #7: DEVICE_PSSETSHADERRESOURCES_HAZARD] Exception thrown at 0x773224C2 in lowlatencymrcsample.exe: Microsoft C++ exception: _com_error at memory location 0x0DB4C380. D3D11 CORRUPTION: ID3D11DeviceContext::UpdateSubresource: Two threads were found to be executing functions associated with the same Device[Context] at the same time. This will cause corruption of memory. Appropriate thread synchronization needs to occur external to the Direct3D API (or through the ID3D10Multithread interface). 4668 and 1584 are the implicated thread ids. [ MISCELLANEOUS CORRUPTION #28: CORRUPTED_MULTITHREADING] Exception thrown at 0x773224C2 (KernelBase.dll) in lowlatencymrcsample.exe: 0x0000087D (parameters: 0x00000000, 0x0470DA8C, 0x0470CEC4). Exception thrown: 'System.Runtime.InteropServices.SEHException' in Assembly-CSharp.dll The program '[4836] lowlatencymrcsample.exe' has exited with code 0 (0x0).

The exception triggers at line 564 in PlaybackEngine.cpp, on the spBuffer->Unlock() instruction.

It appears when i set the Hololens to be the playback device while Unity does the capture of the local PC webcam. Streaming from Hololens to Unity seems to be stable enough but streaming from PC to Hololens is very unstable and lead to this error 95% of the time with 5% working properly enough (lagging though, probably because of those warnings).

Would like to figure out myself what is wrong with the critical section and D3D11 multithreading issue but since you worked on this you probably have a better idea of what's going on there.

Thanks for your support.

[ghost]

That was not a scenario that was in mind for this component. The design of the PlaybackEngine was to support direct access to the IMFSample which allows you get the embedded matrix data from Hololens. This is a slow copy, given you are trying to playback on the HoloLens, since the Unity pipeline own the D3D11Device/Context it may not be in a state to allow the operation to happen or is taking to long.

Since your desktop capture will not contain camera orientation, you should replace the PlaybackEngine code with one that uses a IMFMediaEngine. You can create a standalone, media foundation pipeline to use the same network source and its own D3D11Device that is independent of Unity. You then create a shared texture resource that will be used by Unity, but it handled by MediaFoundation(see TransferVideoFrame). This will ensure MediaFoundation keeps all the video data on the GPU and will not starve the graphics pipeline in Unity.

There is a sample in the Windows samples that demonstrates some of this. https://code.msdn.microsoft.com/windowsapps/Media-Engine-Playback-ce1c82f0#content

The key will be how you create your texture:

    UINT resetToken;
    ComPtr<IMFDXGIDeviceManager> spDXGIManager;
    LogErrorAndThrow(
        MFCreateDXGIDeviceManager(&resetToken, &spDXGIManager)
        );

    LogErrorAndThrow(
        spDXGIManager->ResetDevice(spAltDevice.Get(), resetToken)
        );

    ComPtr<IMFMediaEngineClassFactory> spFactory;
    LogErrorAndThrow(
        CoCreateInstance(CLSID_MFMediaEngineClassFactory, nullptr, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&spFactory))
        );

    ComPtr<IMFAttributes> spAttributes;
    LogErrorAndThrow(
        MFCreateAttributes(&spAttributes, 1)
        );

    LogErrorAndThrow(
        spAttributes->SetUnknown(MF_MEDIA_ENGINE_DXGI_MANAGER, spDXGIManager.Get())
        );

    LogErrorAndThrow(
        spAttributes->SetUINT32(MF_MEDIA_ENGINE_VIDEO_OUTPUT_FORMAT, _textureDesc.Format)
        );

    // create texture on the unity device
    LogErrorAndThrow(
        spDevice->CreateTexture2D(&outputTextureDesc, nullptr, &_sharedOutputTexture)
        );

    LogErrorAndThrow(
        spDevice->CreateShaderResourceView(_sharedOutputTexture.Get(), nullptr, &_outputTextureSRV)
        );

// later in the timer function:
            // get the dxgi resource
            ComPtr<IDXGIResource> outputTextureDXGIResource;
            HRESULT hr = _sharedOutputTexture.As(&outputTextureDXGIResource);
            if (FAILED(hr))
            {
                return;
            }

            HANDLE hOutputTextureSharedHandle;
            hr = outputTextureDXGIResource->GetSharedHandle(&hOutputTextureSharedHandle);
            if (FAILED(hr))
            {
                return;
            }

            hr = spAltDevice->OpenSharedResource(hOutputTextureSharedHandle, IID_PPV_ARGS(&_spMediaTexture));
            if (FAILED(hr))
            {
                return;
            }

        LogErrorAndThrow(
            _spMediaEngine->TransferVideoFrame(_spMediaTexture.Get(), nullptr, &rcTarget, &bkgColor)
            );

``