MultiplayerSample Crash on Quit: Atom Material

[mcphedar]

Describe the bug Calling 'quit' results in a bad shuitdown due to atom material cleanup

Steps to reproduce Steps to reproduce the behavior:

1. Run the Multiplayer Sample
2. Quit out using the esc menu

Expected behavior The MPS should quit cleanly

Actual behavior A clear and concise description of what actually happened.

Assets required A crash occurs - crash dmp -

Exception Description: EXCEPTION_ACCESS_VIOLATION, Attempt to read from address 0x18446744073709551615

The memory could not be "read"
Call Stack Trace:
25) 00007FFBF5B80327 (KERNELBASE) : UnhandledExceptionFilter
24) 00007FFBF8135530 (ntdll) : memset
23) 00007FFBF811C876 (ntdll) : _C_specific_handler
22) 00007FFBF813241F (ntdll) : _chkstk
21) 00007FFBF80E14A4 (ntdll) : RtlRaiseException
20) 00007FFBF8130F4E (ntdll) : KiUserExceptionDispatcher
19) D:\github\o3de\Code\Framework\AzCore\AzCore\Memory\HphaAllocator.cpp (1996) : AZ::HphaSchemaBase<0>::HpAllocator::tree_free
18) D:\github\o3de\Code\Framework\AzCore\AzCore\Memory\HphaAllocator.cpp (927) : AZ::HphaSchemaBase<0>::HpAllocator::free
17) 00007FF6E877BDD0 (MultiplayerSample.GameLauncher) : AZStd::Internal::hash_table_storage<AZStd::Internal::UnorderedSetTableTraits<AZ::EntityId,AZStd::hash<AZ::EntityId>,AZStd::equal_to<AZ::EntityId>,AZStd::allocator,0>,1>::~hash_table_storage<AZStd::Inter
16) D:\github\o3de\Code\Framework\AzCore\AzCore\std\createdestroy.h (59) : AZStd::Internal::destroy<AZ::RPI::ShaderCollection::Item *,AZ::RPI::ShaderCollection::Item,0>::range
15) D:\github\o3de\Code\Framework\AzCore\AzCore\std\containers\vector.h (338) : AZStd::vector<AZ::RPI::ShaderCollection::Item,AZStd::allocator>::~vector<AZ::RPI::ShaderCollection::Item,AZStd::allocator>
14) 00007FF6EDA3BC9E (MultiplayerSample.GameLauncher) : AZ::RPI::Material::`scalar deleting destructor'
13) D:\github\o3de\Code\Framework\AzCore\AzCore\std\function\function_template.h (685) : AZStd::function<void __cdecl(AZ::RPI::Material *)>::operator()
12) D:\github\o3de\Code\Framework\AtomCore\AtomCore\Instance\InstanceDatabase.h (444) : AZ::Data::InstanceDatabase<AZ::RPI::Material>::ReleaseInstance
11) D:\github\o3de\Code\Framework\AtomCore\AtomCore\Instance\InstanceData.cpp (64) : AZ::Data::InstanceData::release
10) D:\github\o3de\Gems\Atom\Feature\Common\Code\Source\Mesh\MeshFeatureProcessor.cpp (917) : AZ::Render::ModelDataInstance::DeInit
9) D:\github\o3de\Gems\Atom\Feature\Common\Code\Source\Mesh\MeshFeatureProcessor.cpp (425) : AZ::Render::MeshFeatureProcessor::SetMaterialAssignmentMap
8) D:\github\o3de\Code\Framework\AzCore\AzCore\EBus\Internal\BusContainer.h (146) : AZ::Internal::EBusContainer<AZ::Render::MaterialComponentNotifications,AZ::Render::MaterialComponentNotifications,1,1>::Dispatcher<AZ::EBus<AZ::Render::MaterialComponentNoti
7) D:\github\o3de\Gems\AtomLyIntegration\CommonFeatures\Code\Source\Material\MaterialComponentController.cpp (134) : AZ::Render::MaterialComponentController::Deactivate
6) D:\github\o3de\Code\Framework\AzCore\AzCore\Component\Entity.cpp (242) : AZ::Entity::Deactivate
5) D:\github\o3de\Code\Framework\AzCore\AzCore\Component\Entity.cpp (111) : AZ::Entity::Reset
4) D:\github\o3de\Code\Framework\AzCore\AzCore\Component\Entity.cpp (104) : AZ::Entity::~Entity
3) 00007FF6EE7A6934 (MultiplayerSample.GameLauncher) : AZ::Entity::`scalar deleting destructor'
2) D:\github\o3de\Code\Framework\AzCore\AzCore\Component\ComponentApplication.cpp (834) : AZ::ComponentApplication::Destroy
1) D:\github\o3de\Code\Framework\AzFramework\AzFramework\Application\Application.cpp (273) : AzFramework::Application::Stop 

[lmbr-pip]

@mcphedar Can you attach your commit ids to this report and information about your build (debug, profile etc)?

[VickyAtAZ]

@mbalfour-amzn and I couldn't repro the crash with this call stack. But we have seen two other crashes which happened randomly when exit the game launcher for client.

The crashes happened with development branch with o3de commit: 4b65521188a2a0beb399eca7b692216d2fe3d209 and MPS commit: 15da465c55834cbe5033c0c3aa15e405767d2823

One is

>   Atom_RHI_DX12.Private.dll!D3D12MA::Allocation::ReleaseThis() Line 9687  C++
    Atom_RHI_DX12.Private.dll!D3D12MA::IUnknownImpl::Release() Line 9593    C++
    [Inline Frame] Atom_RHI_DX12.Private.dll!AZStd::IntrusivePtrCountPolicy<ID3D12Heap>::release(ID3D12Heap * p) Line 91    C++
    [Inline Frame] Atom_RHI_DX12.Private.dll!AZStd::intrusive_ptr<ID3D12Heap>::{dtor}() Line 119    C++
    [Inline Frame] Atom_RHI_DX12.Private.dll!AZStd::Internal::destroy<AZStd::intrusive_ptr<ID3D12Heap> *,AZStd::intrusive_ptr<ID3D12Heap>,0>::range(AZStd::intrusive_ptr<ID3D12Heap> * first, AZStd::intrusive_ptr<ID3D12Heap> * last) Line 61  C++
    Atom_RHI_DX12.Private.dll!AZStd::vector<AZStd::intrusive_ptr<ID3D12Heap>,AZStd::allocator>::~vector<AZStd::intrusive_ptr<ID3D12Heap>,AZStd::allocator>() Line 336   C++
    Atom_RHI_DX12.Private.dll!AZ::RHI::ObjectCollector<AZ::DX12::D3d12maReleaseQueueTraits>::Collect(bool forceFlush) Line 240  C++
    [Inline Frame] Atom_RHI_DX12.Private.dll!AZ::RHI::ObjectCollector<AZ::DX12::D3d12maReleaseQueueTraits>::Shutdown() Line 134 C++
    Atom_RHI_DX12.Private.dll!AZ::DX12::Device::ShutdownInternal() Line 175 C++
    Atom_RHI_DX12.Private.dll!AZ::RHI::Device::Shutdown() Line 112  C++
    Atom_RPI.Private.dll!AZ::RHI::Object::release() Line 61 C++
    [Inline Frame] Atom_RPI.Private.dll!AZStd::IntrusivePtrCountPolicy<AZ::RHI::Device>::release(AZ::RHI::Device *) Line 48 C++
    [Inline Frame] Atom_RPI.Private.dll!AZStd::intrusive_ptr<AZ::RHI::Device>::{dtor}() Line 119    C++
    [Inline Frame] Atom_RPI.Private.dll!AZStd::Internal::destroy<AZStd::intrusive_ptr<AZ::RHI::Device> *,AZStd::intrusive_ptr<AZ::RHI::Device>,0>::single(AZStd::intrusive_ptr<AZ::RHI::Device> *) Line 68  C++
    [Inline Frame] Atom_RPI.Private.dll!AZStd::vector<AZStd::intrusive_ptr<AZ::RHI::Device>,AZStd::allocator>::pop_back() Line 580  C++
    Atom_RPI.Private.dll!AZ::RHI::RHISystem::Shutdown() Line 214    C++
    Atom_RPI.Private.dll!AZ::RPI::RPISystem::Shutdown() Line 163    C++
    Atom_RPI.Private.dll!AZ::RPI::RPISystemComponent::Deactivate() Line 143 C++
    [Inline Frame] MultiplayerSample.GameLauncher.exe!AZ::Entity::DeactivateComponent(AZ::Component & component) Line 400   C++
    MultiplayerSample.GameLauncher.exe!AZ::ModuleManager::DeactivateEntities() Line 227 C++
    MultiplayerSample.GameLauncher.exe!AZ::ComponentApplication::Destroy() Line 898 C++
    MultiplayerSample.GameLauncher.exe!AzFramework::Application::Stop() Line 273    C++
    MultiplayerSample.GameLauncher.exe!O3DELauncher::Run(const O3DELauncher::PlatformMainInfo & mainInfo) Line 602  C++
    MultiplayerSample.GameLauncher.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Line 29    C++

and the other one is

    MultiplayerSample.GameLauncher.exe![thunk]:AzFramework::ApplicationLifecycleEvents::`vcall'{80,{flat}}' }'()    C++
>   [Inline Frame] MultiplayerSample.GameLauncher.exe!AZStd::Internal::INVOKE(void(AZ::ComponentDescriptor::*)() &) Line 177    C++
    [Inline Frame] MultiplayerSample.GameLauncher.exe!AZStd::invoke(void(AZ::ComponentDescriptor::*)() &) Line 54   C++
    [Inline Frame] MultiplayerSample.GameLauncher.exe!AZ::EBusEventProcessingPolicy::Call(void(AZ::ComponentDescriptor::*)() &) Line 437    C++
    MultiplayerSample.GameLauncher.exe!AZ::Internal::EBusContainer<AZ::ComponentDescriptor,AZ::ComponentDescriptorBusTraits,1,0>::Dispatcher<AZ::EBus<AZ::ComponentDescriptor,AZ::ComponentDescriptorBusTraits>>::Broadcast<void (__cdecl AZ::ComponentDescriptor::*)(void)>(void(AZ::ComponentDescriptor::*)() && func) Line 968   C++
    MultiplayerSample.GameLauncher.exe!AZ::ComponentApplication::Destroy() Line 933 C++
    MultiplayerSample.GameLauncher.exe!AzFramework::Application::Stop() Line 273    C++
    MultiplayerSample.GameLauncher.exe!O3DELauncher::Run(const O3DELauncher::PlatformMainInfo & mainInfo) Line 602  C++
    MultiplayerSample.GameLauncher.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Line 29    C++

[drakeredwind01]

ask:

should we add tag:

needs-priority triage/needs-information status/needs-testing

[mbalfour-amzn]

These should be fixed by these: https://github.com/o3de/o3de/pull/15880 https://github.com/o3de/o3de/pull/15868

[AMZN-Gene]

Closed https://github.com/o3de/o3de/pull/15880 https://github.com/o3de/o3de/pull/15868