_com_error at memory location 0x0000006A74D6D680 when calling generator.ComputeLogits()

[tranlm]

Describe the bug I've tried to copy the HelloPhi example for loading and calling phi-3.5 in DirectML for my application. When I get to the generator.ComputeLogits() part, I get a ton (e.g. hundreds) of the following memory errors:

Exception thrown at 0x00007FFFF9ED6D9A in <MyApplicationsName>.exe: Microsoft C++ exception: _com_error at memory location 0x0000006A74D6D680.

After a few minutes, the inference engine starts generating the text.

To Reproduce Steps to reproduce the behavior:

1. Copy the DirectML version of the HelloPhi example as a new Blank App, Packaged with Windows Application Packing Project (WinUI 3 in Desktop) app with framework net8.0-windows10.0.19041.0 (n.b. I'm using WindowsSdkPackageVersion 10.0.19041.38).
2. Download and convert phi-3.5 to DirectML (or rely on the onnx dml verison at microsoft/Phi-3-mini-128k-instruct-onnx on huggingface - I've tried both models).
3. Run the app with the downloaded model (making sure to break when c++ exceptions are thrown.
4. See error

Expected behavior No memory errors to show and to start generating text immediately (rather than after 3-5 minutes of the memory exceptions).

Screenshots

Desktop (please complete the following information):

• OS: Windows 11

[tranlm]

Quick follow-up. I tried updating to the most recent Windows SDK (10.0.26100.0) and ran the app again. Now I'm getting a slightly different error (coming out of D3D12Core rather than KernelBase).

[baijumeswani]

@PatriceVignola would you happen to know where this error might be coming from?

[tranlm]

I’ve been in touch with Microsoft developers and we’ve pinpointed that this is due to our packaging the build separately in our project. However, root cause is still unclear.

On Wed, Oct 9, 2024 at 11:56 AM Baiju Meswani @.***> wrote:

@PatriceVignola https://github.com/PatriceVignola would you happen to know where this error might be coming from?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/onnxruntime-genai/issues/886#issuecomment-2402839349, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAROROAIAIAR4R7WX24BR53Z2VN3HAVCNFSM6AAAAABN74S4ECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBSHAZTSMZUHE . You are receiving this because you authored the thread.Message ID: @.***>

[tranlm]

In case it helps, here is the call stack. I tried to load all of the pdb's I could find, but don't know where to locate the ones for onnxruntime.dll or onnxruntime-genai.dll. Please point me to it if you know and I'll update the call stack accordingly. Thanks.

    KernelBase.dll!RaiseException() Unknown
    D3D12Core.dll!_CxxThrowException()  Unknown
    D3D12Core.dll!ThrowFailure(long)    Unknown
    D3D12Core.dll!CMetaCommand::FinalConstruct(struct CMetaCommand::TConstructorArgs const &)   Unknown
    D3D12Core.dll!CLayeredObject<class CMetaCommand>::CreateInstance(struct CMetaCommand::TConstructorArgs const &,void *,void *,struct _GUID const &,void * *) Unknown
    D3D12Core.dll!CDevice::CreateLayeredChild() Unknown
    D3D12Core.dll!NDXGI::CDevice::CreateLayeredChild(unsigned int,void const *,unsigned __int64,struct ID3D12LayeredUseCounted *,struct _GUID const &,void * *) Unknown
    D3D12.dll!NOutermost::CDevice::CreateLayeredChild(unsigned int,void const *,unsigned __int64,struct ID3D12LayeredUseCounted *,struct _GUID const &,void * *)    Unknown
    D3D12Core.dll!CDevice::CreateMetaCommand()  Unknown
    DirectML.dll!MetaCommand::TryCreate(class DmlDevice *,struct _GUID const &,class gsl::span<enum std::byte const ,-1>,class std::optional<struct NPUMetacommandQueriedRequirements>) Unknown
    DirectML.dll!QuantizedGemmMetaCommand::TryCreateLatest(class DmlDevice *,enum DML_EXECUTION_FLAGS,bool,struct DmlQuantizedGemmOperatorDesc const &) Unknown
    DirectML.dll!TryCreateWithVersionFallback<QuantizedGemmMetaCommand,DmlQuantizedGemmOperatorDesc>()  Unknown
    DirectML.dll!DmlMetacommandGpu::TryCreateQuantizedGEMM(class DmlOperator *,enum DML_EXECUTION_FLAGS,struct DmlQuantizedGemmOperatorDesc const &)    Unknown
    DirectML.dll!DmlMetaCommand::TryCreateQuantizedGEMM(class DmlOperator *,enum DML_EXECUTION_FLAGS,struct DmlQuantizedGemmOperatorDesc const &)   Unknown
    DirectML.dll!DmlQuantizedGemmOperator::Compile(enum DML_EXECUTION_FLAGS)    Unknown
    DirectML.dll!DmlDevice::CompileOperator(struct IDMLOperator *,enum DML_EXECUTION_FLAGS,struct _GUID const &,void * *)   Unknown
    DirectML.dll!OperatorCache::GetNonInitializableCompiledOperator(struct AbstractOperatorDesc const &,enum DML_EXECUTION_FLAGS)   Unknown
    DirectML.dll!MLGraph::Compilation::LatestDmlRuntimeDeviceProfile::TryGetCompiledOperator(struct AbstractOperatorDesc const &,enum DML_EXECUTION_FLAGS)  Unknown
    DirectML.dll!MLGraph::Compilation::LatestDmlRuntimeDeviceProfile::GetResourceRequirements(struct AbstractOperatorDesc const &,bool,bool,bool)   Unknown
    DirectML.dll!MLGraph::Passes::AssignTemporaryAndPersistentTensors::ProcessNode(class std::shared_ptr<class MLGraph::CompilerIR::Node> const &,bool) Unknown
    DirectML.dll!MLGraph::Passes::AssignTemporaryAndPersistentTensors::Execute(class gsl::span<class std::shared_ptr<class MLGraph::CompilerIR::Node>,-1>,class MLGraph::ICompileLogger *)  Unknown
    DirectML.dll!MLGraph::Compilation::PassManager::ExecutePasses(struct CommonLogging::ILogger *,enum MLGraph::CompileLogMask,class MLGraph::ICompileLogger *) Unknown
    DirectML.dll!MLGraph::Compilation::LegacyCompiler::GraphCompiler::CompileGraph(class std::shared_ptr<class MLGraph::Compilation::LegacyCompiler::IDeviceProfile>,class std::shared_ptr<class MLGraph::ICompileLogger>,class gsl::span<class std::shared_ptr<class MLGraph::CompilerIR::Node>,-1>,class gsl::span<class std::shared_ptr<class MLGraph::CompilerIR::Node>,-1>,struct DMLExecutionPlan::ExecutionPlanContext *,bool,bool,bool,bool)    Unknown
    DirectML.dll!LegacyCompileGraph(struct DmlGraphDesc const *,class std::shared_ptr<class MLGraph::Compilation::LegacyCompiler::IDeviceProfile>,class std::shared_ptr<class MLGraph::ICompileLogger>,struct DMLExecutionPlan::ExecutionPlanContext *,bool,bool,bool,bool,bool,bool)   Unknown
    DirectML.dll!DmlDevice::CompileGraph(struct DmlGraphDesc const *,class std::shared_ptr<class IOperatorCache>,bool,bool,bool,bool,struct DMLExecutionPlan::ExecutionPlanContext *)   Unknown
    DirectML.dll!DmlDevice::CompileGraph(struct DmlGraphDesc const *,enum DML_EXECUTION_FLAGS,class DmlOperator *,class std::shared_ptr<struct DmlConstantInputBindings>)   Unknown
    DirectML.dll!DmlDevice::CompileGraph(struct DML_GRAPH_DESC const *,enum DML_EXECUTION_FLAGS,struct _GUID const &,void * *)  Unknown
    onnxruntime.dll!00007ffff426e25b()  Unknown
    onnxruntime.dll!00007ffff430494d()  Unknown
    onnxruntime.dll!00007ffff48b1fdd()  Unknown
    onnxruntime.dll!00007ffff48db934()  Unknown
    onnxruntime.dll!00007ffff48dad07()  Unknown
    onnxruntime.dll!00007ffff48db162()  Unknown
    onnxruntime.dll!00007ffff48db0cb()  Unknown
    onnxruntime.dll!00007ffff48dbe0f()  Unknown
    onnxruntime.dll!00007ffff48dad07()  Unknown
    onnxruntime.dll!00007ffff48b3272()  Unknown
    onnxruntime.dll!00007ffff48b2859()  Unknown
    onnxruntime.dll!00007ffff487d699()  Unknown
    onnxruntime.dll!00007ffff487db9e()  Unknown
    onnxruntime.dll!00007ffff487dc93()  Unknown
    onnxruntime.dll!00007ffff41a87ec()  Unknown
    onnxruntime.dll!00007ffff41a9d47()  Unknown
    onnxruntime.dll!00007ffff4147199()  Unknown
    onnxruntime-genai.dll!00007ffffcc9e62a()    Unknown
    onnxruntime-genai.dll!00007ffffcc8f06b()    Unknown
    onnxruntime-genai.dll!00007ffffcc7fa1e()    Unknown
    onnxruntime-genai.dll!00007ffffccb29d9()    Unknown
    [Managed to Native Transition]  
    Microsoft.ML.OnnxRuntimeGenAI.dll!Microsoft.ML.OnnxRuntimeGenAI.Generator.ComputeLogits()   Unknown
>   Project1.dll!Project1.SLMRunner.InferStreamingAsync(string prompt, System.Threading.CancellationToken ct) Line 67   C#