gameserver panic in Profile.cpp when start up occasionally

[tiancaiamao]

GameServer::init() : ObjectManager Initialization Success...
GameServer::init() : ThreadManager Initialization Success...
GameServer::init() : PacketFactoryManager Initialization Success...
GameServer::init() : PacketValidator Initialization Success...
GameServer::init() : LoginServerManager Initialization Success...
GameServer::init() : SharedServerManager Initialization Success...
GameServer::init() : MPacketManager Initialization Success...
GameServer::init() : MPlayerManager Initialization Success...
MAX SERVER GROUP = 1
WorldID:1 ServerGroupID:0 follows0
GameServerInfoManager(
GameServerInfo(ServerID: 1,Nickname:game1,IP: 192.168.0.16,TCPPort:9998,UDPPort:9997,GroupID:0,WorldID:1,ServerStat:0)
)EMPTY)
GameServer::init() : GameServerInfoManager Initialization Success...
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 THREAD CONNECT UIIRIBUTION DB 
 TID Number = -199259776
******************************************************
GameServer::init() : ClientManager Initialization Success...
>>> GAME SERVER INITIALIZATION SUCCESS...
>>> STARTING THREAD MANAGER...
[New Thread 0x7ffff41a4700 (LWP 873606)]
[New Thread 0x7ffff39a3700 (LWP 873607)]
>>> STARTING LOGIN SERVER MANAGER...
[New Thread 0x7ffff31a2700 (LWP 873608)]
>>> STARTING SHARED SERVER MANAGER...
[New Thread 0x7ffff29a1700 (LWP 873609)]
[New Thread 0x7ffff21a0700 (LWP 873610)]
>>> STARTING MOFUS PLAYER MANAGER...
Init GDR Lair Manager
2022.09.26-14:00:00:000�� ���巹 ���� �ٽ� ����
2022.09.26-14:00:00:000�� ���巹 ���� �ٽ� ����
[New Thread 0x7ffff199f700 (LWP 873611)]
>>> ALL INITIALIZATIONS ARE COMPLETED SUCCESSFULLY.
>>> STARTING ClientManager->start() INFINITE LOOP...
Adding TID connection BEGINAdding TID connection BEGIN

Adding TID connection END
************************************************************************
************************************************************************
************************************************************************
************************************************************************
OPEN LOGIN DB
************************************************************************
************************************************************************
************************************************************************
Adding TID connection END
******************************************************
 THREAD CONNECT DB 
******************************************************
Adding TID connection BEGINAdding TID connection BEGIN

Adding TID connection BEGIN
Adding TID connection ENDAdding TID connection END

Adding TID connection END******************************************************

 Mofus THREAD CONNECT DB 
******************************************************
******************************************************
 THREAD CONNECT DB 
******************************************************
************************************************************************
OPEN LOGIN DB
************************************************************************
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 GDR Lair THREAD CONNECT DB 
******************************************************
connection to sharedserver established
Send:438 GSRequestGuildInfo
Receive:SGGuildInfo()
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 THREAD CONNECT UIIRIBUTION DB 
 TID Number = -207997184
******************************************************
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 THREAD CONNECT UIIRIBUTION DB 
 TID Number = -199604480
******************************************************

Thread 7 "gameserver" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff199f700 (LWP 873611)]

From the panic stack, it seems that the hash table is not initialized, which should be done in ZoneGroupThread.cpp initProfileEx()

Maybe this is related to parallel, the execution order of the threads is undetermined, so maybe ZoneGroupThread has not run when GDRLairManager::run goes to endProfile...

Or, maybe a data race?

(gdb) bt
#0  0x0000555555fcf700 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_bucket_index (this=0xbb80, __k="ZPM_QUERY", __c=10880332307479813206)
    at /usr/include/c++/9/bits/hashtable.h:676
#1  0x0000555555fcf373 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0xbb80, __k="ZPM_QUERY") at /usr/include/c++/9/bits/hashtable.h:1453
#2  0x0000555555fcf211 in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > >::find (this=0xbb80, __x="ZPM_QUERY") at /usr/include/c++/9/bits/unordered_map.h:922
#3  0x0000555557ed0a2f in ProfileSampleSet::endProfile (this=0x0, name="ZPM_QUERY") at Profile.cpp:182
#4  0x0000555557e92398 in Statement::executeQuery (this=0x7fffd4007e80) at Statement.cpp:139
#5  0x0000555557e926d3 in Statement::executeQueryString (this=0x7fffd4007e80, sqlStatement="SELECT 1") at Statement.cpp:160
#6  0x0000555557e9c0ab in DatabaseManager::executeDummyQuery (this=0x55556108e9f0, pConnection=0x7fffd4000b60) at DatabaseManager.cpp:496
#7  0x00005555563aeae0 in GDRLairManager::run (this=0x55555dcd8520 <GDRLairManager::Instance()::theInstance>) at GDRLairManager.cpp:99
#8  0x0000555557ec3b9f in start_routine (derivedThread=0x55555dcd8540 <GDRLairManager::Instance()::theInstance+32>) at Thread.cpp:225
#9  0x00007ffff7f95609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#10 0x00007ffff7092163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) 
#0  0x0000555555fcf700 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_bucket_index (this=0xbb80, __k="ZPM_QUERY", __c=10880332307479813206)
    at /usr/include/c++/9/bits/hashtable.h:676
#1  0x0000555555fcf373 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::find (this=0xbb80, __k="ZPM_QUERY") at /usr/include/c++/9/bits/hashtable.h:1453
#2  0x0000555555fcf211 in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, int> > >::find (this=0xbb80, __x="ZPM_QUERY") at /usr/include/c++/9/bits/unordered_map.h:922
#3  0x0000555557ed0a2f in ProfileSampleSet::endProfile (this=0x0, name="ZPM_QUERY") at Profile.cpp:182
#4  0x0000555557e92398 in Statement::executeQuery (this=0x7fffd4007e80) at Statement.cpp:139
#5  0x0000555557e926d3 in Statement::executeQueryString (this=0x7fffd4007e80, sqlStatement="SELECT 1") at Statement.cpp:160
#6  0x0000555557e9c0ab in DatabaseManager::executeDummyQuery (this=0x55556108e9f0, pConnection=0x7fffd4000b60) at DatabaseManager.cpp:496
#7  0x00005555563aeae0 in GDRLairManager::run (this=0x55555dcd8520 <GDRLairManager::Instance()::theInstance>) at GDRLairManager.cpp:99
#8  0x0000555557ec3b9f in start_routine (derivedThread=0x55555dcd8540 <GDRLairManager::Instance()::theInstance+32>) at Thread.cpp:225
#9  0x00007ffff7f95609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#10 0x00007ffff7092163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[tiancaiamao]

Another stack:

GameServer::init() : ClientManager Initialization Success...
>>> GAME SERVER INITIALIZATION SUCCESS...
>>> STARTING THREAD MANAGER...
[New Thread 0x7ffff41a4700 (LWP 878085)]
[New Thread 0x7ffff39a3700 (LWP 878086)]
>>> STARTING LOGIN SERVER MANAGER...
[New Thread 0x7ffff31a2700 (LWP 878087)]
>>> STARTING SHARED SERVER MANAGER...
[New Thread 0x7ffff29a1700 (LWP 878088)]
[New Thread 0x7ffff21a0700 (LWP 878089)]
>>> STARTING MOFUS PLAYER MANAGER...
Init GDR Lair Manager
2022.09.26-14:00:00:000�� ���巹 ���� �ٽ� ����
2022.09.26-14:00:00:000�� ���巹 ���� �ٽ� ����
[New Thread 0x7ffff199f700 (LWP 878090)]
>>> ALL INITIALIZATIONS ARE COMPLETED SUCCESSFULLY.
>>> STARTING ClientManager->start() INFINITE LOOP...
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 THREAD CONNECT DB 
******************************************************
Adding TID connection BEGIN
Adding TID connection BEGIN
Adding TID connection END
************************************************************************
************************************************************************Adding TID connection END
******************************************************
 THREAD CONNECT DB 
******************************************************

************************************************************************
************************************************************************
OPEN LOGIN DB
************************************************************************
************************************************************************
************************************************************************
Adding TID connection BEGIN
Adding TID connection END
************************************************************************
OPEN LOGIN DB
************************************************************************
connection to sharedserver established
Send:438[0,0] GSRequestGuildInfo
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 GDR Lair THREAD CONNECT DB 
******************************************************
Adding TID connection BEGIN
Receive:SGGuildInfo()Adding TID connection END

******************************************************
 THREAD CONNECT UIIRIBUTION DB 
 TID Number = Adding TID connection BEGIN-207997184
******************************************************

Adding TID connection END
******************************************************
 THREAD CONNECT UIIRIBUTION DB 
 TID Number = -199604480
******************************************************
Adding TID connection BEGIN
Adding TID connection END
******************************************************
 Mofus THREAD CONNECT DB 
******************************************************
AssertionError : 
Assertion Failed : Profile.cpp : 187 : void ProfileSampleSet::endProfile(const string&)false at Mon Sep 26 09:19:23 2022

==============================================================================
UNHANDLED EXCEPTION OCCURED
==============================================================================

Thread 2 "gameserver" received signal SIGABRT, Aborted.

This time it happens during ZoneGroupThread init

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff6f95859 in __GI_abort () at abort.c:79
#2  0x0000555555cab733 in handleUnhandledException () at main.cpp:39
#3  0x00007ffff737b38c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff737b3f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff737b6fd in __cxa_rethrow () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000555555fa59d6 in ZoneGroupThread::run (this=0x55556b967fe0) at ZoneGroupThread.cpp:191
#7  0x0000555557ec3b9f in start_routine (derivedThread=0x55556b967fe0) at Thread.cpp:225
#8  0x00007ffff7f95609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00007ffff7092163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[NEW-BOY1]

@tiancaiamao Their associated ZoneGroupThread.cpp? or void ProfileSampleSet::endProfile This error appears in the new task? If you confirmation of the old version also has this problem

[tiancaiamao]

@tiancaiamao Their associated ZoneGroupThread.cpp? or void ProfileSampleSet::endProfile This error appears in the new task? If you confirmation of the old version also has this problem

void ProfileSampleSet::endProfile

This is where the panic happen.

The stack in ZoneGroupThread.cpp is the catch's code path of the try-catch, not where the panic really happen.

[NEW-BOY1]

AssertionError : Assertion Failed : Profile.cpp : 187 : void ProfileSampleSet::endProfile

void ProfileSampleSet::endProfile(const string& name) { unordered_map<string, int>::iterator itr = m_NameMap.find(name);

// 그런 이름을 가진 프로파일 샘플이 존재하지 않는다면 에러다...
if (itr == m_NameMap.end())   //  Who started it？
{
    Assert(false);   // 187   This is where ？
}

@tiancaiamao

[Supacek]

it can happened if the maximum amount of profiles was exceeded.

[Supacek]

[Supacek]

const int MAX_PROFILE_SAMPLES = 99999;

[tiancaiamao]

const int MAX_PROFILE_SAMPLES = 99999;

I'll take a try