Open FaanMaario opened 8 years ago
To help us resolve this, you can create a memory dump when your server is at high usage and upload it and send a link.
How to create a memory dump:
pocketmine.yml
and set debug.commands
to true
/status
to see how much memory your server is using./dumpmemory
. This will create a memory dump in your server folder under /memory_dumps
.Thanks for your cooperation.
Something attempted to allocate 576 MB for a single object? WTF?
@dktapps Yeah I'm surprised too.
Here the dumpmemory folder I just created a minute ago @dktapps -> https://drive.google.com/file/d/0B1zyNGYeW8RRaUlNRnUyVmc5UE0/view?usp=sharing
Same here. I was experiencing a strange error in the past days: Genisys was not responding to any command input at console, players couldn't execute any command, no block updates, other players failed to join the server. The server kept freezing for about 10 minutes, then Genisys process was eventually killed by either system or itself.
The freeze always happened suddenly when RAM usage seemed to be normal, main thread memory was only around 100-150 MB. The most mysterious thing is: No crash dumps generated.
I used top
in CentOS to observe the PHP process when Genisys froze: CPU usage approached 100% (98%-99%), RAM usage continuously increased until Genisys ate up all available memory.
Genisys version: 96940b834ebf8c4040cc453af5f01d5642987ac5 (By the way, the PHAR file I used was the one I produced using DevTools with source code on GitHub. Does it matter?) PHP: 7.0.3 with OpenSSL plugin Server OS: CentOS 7.2.1511
Because there wasn't any omen, I couldn't grab any memory dump. But this morning, after I decreased memory limit of Genisys, I successfully obtained 3 same crash dumps (Exactly the same excluding time stamps, server uptime and players online, even bytes tried to allocate was the same, 150994944.
Genisys Crash Dump Sun Aug 28 10:42:30 CST 2016
Error: Allowed memory size of 805306368 bytes exhausted (tried to allocate 150994944 bytes)
File: /src/pocketmine/level/Level
Line: 1397
Type: E_ERROR
Code:
[1388] }
[1389]
[1390] $block = clone $this->blockStates[$fullState & 0xfff];
[1391]
[1392] $block->x = $pos->x;
[1393] $block->y = $pos->y;
[1394] $block->z = $pos->z;
[1395] $block->level = $this;
[1396]
[1397] return $this->blockCache[$index] = $block;
[1398] }
[1399]
[1400] public function updateAllLight(Vector3 $pos){
[1401] $this->updateBlockSkyLight($pos->x, $pos->y, $pos->z);
[1402] $this->updateBlockLight($pos->x, $pos->y, $pos->z);
[1403] }
[1404]
[1405] public function updateBlockSkyLight($x, $y, $z){
[1406] //TODO
[1407] }
Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)
Genisys version: 0000000000000000000000000000000000000000 [Protocol 82; API 2.0.0]
uname -a: Linux VM_158_136_centos 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64
PHP version: 7.0.3
Zend version: 3.0.0
OS : Linux, linux
Server uptime: 17 分 32 秒
Number of loaded worlds: 8
Players online: 2/40
Loaded plugins:
AdvancedMOTD 1.0.1 by Philip Shilling for API(s) 1.12.0
Whois 3 by hoyinm for API(s) 1.12.0
KillHead 1.0.0 by 古董 for API(s) 1.1.0
TimeCommander 2 by LDX for API(s) 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
TeamPVP 1.1.0 by Zzm for API(s) 1.0.0
BlockProtector 1 by luca28pet for API(s) 1.12.0
mcg76_CTF 1.5.3 by minecraftgenius76, MCG76 for API(s) 1.8.0
ZKit 1.0.0 by for API(s) 1.12.0
BannedWhy 1.0.0 by Matt for API(s) 1.0.0
SimpleAuth 1.7.2 by PocketMine Team, shoghicp for API(s) 2.0.0
CommandSigns 2 by Samueljh1 for API(s) 1.0.0
FurnaceIndicator 1.2 by onebone for API(s) 1.6.0
LockPlayer 1.0.0 by Draemon for API(s) 1.0.0
EconomyAPI 2.0.9 by onebone for API(s) 2.0.0
AshMan 1.0.1 by FENGberd,Alcatraz_Du for API(s) 1.0.0
QituShop 1.2.0 by qitu for API(s) 1.0.0
WorldProtect 2.2.0 by aliuly for API(s) 1.10.0, 1.12.0
DontTapTheWhiteTile 0.1.2 by MUedsa for API(s) 1.8.0
SW_svile 0.6dev by svile for API(s) 2.0.0
SearchID 1.0.2 by FransicSteve for API(s) 1.0.0
InvSee 0.0.1 by tschrock (tschrock123@gmail.com) for API(s) 1.0.0
Watchdog 1.2.3 by Leo3418 for API(s) 2.0.0
EconomyLand 2.0.10 by onebone, Chinese (Simplified) translation by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
BanItem 2.2 by LDX for API(s) 1.0.0, 2.0.0
EconomyShop 2.0.7|✘wine汉化 by onebone for API(s) 1.12.0
SubnetBan 1.2.2 by for API(s) 2.0.0
SimpleAuthHelper 2.0.4 by aliuly, Chinese (Simplified) translation by Leo3418 for API(s) 1.12.0
CommandTracker 1.1 by Scott Handley for API(s) 1.10.0
RealMute 3.2.2-ch by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
MagicWE 0.2.4 by thebigsmileXD for API(s) 1.13.0, 2.0.0
EconomySell 2.0.7|✘wine汉化 by onebone for API(s) 1.12.0
DisableFlow 1.1.0 by MUedsa, upgraded by Leo3418L for API(s) 2.0.0
ColorMatch 1.1.2 by CreeperFace for API(s) 1.12.0
NameLimit 1.0.0 by Leo3418 for API(s) 2.0.0
EasyMessages 1.0.0 by Gamecrafter for API(s) 1.12.0
QD 1 by labi for API(s) 1.0.0
RestartMe 1.2.0 by Gamecrafter for API(s) 1.10.0
Prefix 1.2.0 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
FactionsPro 1.3.2 by Tethered_ for API(s) 1.12.0
essentialsTP 1.1.0 by for API(s) 1.12.0
BlockTP 1.0.3 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
KSpleef 1.3.0 by Kevin for API(s) 1.12.0
InventoryTrack 1.0.1 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
EconomyJob 2.0.4 by onebone for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
PocketGuard 2.1.2 by MinecrafterJPN for API(s) 1.10.0
@leo3418 could you also provide a memory dump as per the instructions in my comment above? Thanks.
@dktapps I'll try, since crash may happen at any time without any sign of freeze.
open pocketmine.yml and set cache-chunk = true
@vzlomka76 Thanks, but I tried and that didn't work. @dktapps Here's the memory dump I obtained. I don't know if it helps because it was grabbed when server was running normally.
@Leo3418 strange, to enable this feature, I ate 30 minutes 5 GB of RAM. After switching on the 150 MB of RAM for 24 hours
@Leo3418 and try to remove unwanted plug-ins and leave only the most necessary
Now I have a different crashdump... That's very strange.
Genisys Crash Dump Sun Aug 28 09:01:59 EDT 2016
Error: Out of memory (allocated 998768640) (tried to allocate 4096 bytes)
File: /src/pocketmine/level/Level
Line: 1390
Type: E_ERROR
Code:
[1381] $index = Level::blockHash($pos->x, $pos->y, $pos->z);
[1382] if($cached and isset($this->blockCache[$index])){
[1383] return $this->blockCache[$index];
[1384] }elseif($pos->y >= 0 and $pos->y < 128 and isset($this->chunks[$chunkIndex = Level::chunkHash($pos->x >> 4, $pos->z >> 4)])){
[1385] $fullState = $this->chunks[$chunkIndex]->getFullBlock($pos->x & 0x0f, $pos->y & 0x7f, $pos->z & 0x0f);
[1386] }else{
[1387] $fullState = 0;
[1388] }
[1389]
[1390] $block = clone $this->blockStates[$fullState & 0xfff];
[1391]
[1392] $block->x = $pos->x;
[1393] $block->y = $pos->y;
[1394] $block->z = $pos->z;
[1395] $block->level = $this;
[1396]
[1397] return $this->blockCache[$index] = $block;
[1398] }
[1399]
[1400] public function updateAllLight(Vector3 $pos){
Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)
Genisys version: 0000000000000000000000000000000000000000 [Protocol 82; API 2.0.0]
uname -a: Linux hms01 2.6.32-43-pve #1 SMP Tue Oct 27 09:55:55 CET 2015 x86_64
PHP version: 7.0.10
Zend version: 3.0.0
OS : Linux, linux
Server uptime: 18 minute(s) 37 seconde(s)
Number of loaded worlds: 4
Players online: 4/40
@FaanMaario Me too. Did you decrease the memory limit?
Genisys Crash Dump Sun Aug 28 23:13:32 CST 2016
Error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 4096 bytes)
File: /src/pocketmine/level/Level
Line: 1390
Type: E_ERROR
Code:
[1381] $index = Level::blockHash($pos->x, $pos->y, $pos->z);
[1382] if($cached and isset($this->blockCache[$index])){
[1383] return $this->blockCache[$index];
[1384] }elseif($pos->y >= 0 and $pos->y < 128 and isset($this->chunks[$chunkIndex = Level::chunkHash($pos->x >> 4, $pos->z >> 4)])){
[1385] $fullState = $this->chunks[$chunkIndex]->getFullBlock($pos->x & 0x0f, $pos->y & 0x7f, $pos->z & 0x0f);
[1386] }else{
[1387] $fullState = 0;
[1388] }
[1389]
[1390] $block = clone $this->blockStates[$fullState & 0xfff];
[1391]
[1392] $block->x = $pos->x;
[1393] $block->y = $pos->y;
[1394] $block->z = $pos->z;
[1395] $block->level = $this;
[1396]
[1397] return $this->blockCache[$index] = $block;
[1398] }
[1399]
[1400] public function updateAllLight(Vector3 $pos){
Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)
Genisys version: 0000000000000000000000000000000000000000 [Protocol 82; API 2.0.0]
uname -a: Linux VM_158_136_centos 3.10.0-327.28.2.el7.x86_64 #1 SMP Wed Aug 3 11:11:39 UTC 2016 x86_64
PHP version: 7.0.3
Zend version: 3.0.0
OS : Linux, linux
Server uptime: 27 分 52 秒
Number of loaded worlds: 8
Players online: 2/40
Loaded plugins:
AdvancedMOTD 1.0.1 by Philip Shilling for API(s) 1.12.0
Whois 3 by hoyinm for API(s) 1.12.0
KillHead 1.0.0 by 古董 for API(s) 1.1.0
TimeCommander 2 by LDX for API(s) 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0
TeamPVP 1.1.0 by Zzm for API(s) 1.0.0
BlockProtector 1 by luca28pet for API(s) 1.12.0
mcg76_CTF 1.5.3 by minecraftgenius76, MCG76 for API(s) 1.8.0
ZKit 1.0.0 by for API(s) 1.12.0
BannedWhy 1.0.0 by Matt for API(s) 1.0.0
SimpleAuth 1.7.2 by PocketMine Team, shoghicp for API(s) 2.0.0
CommandSigns 2 by Samueljh1 for API(s) 1.0.0
FurnaceIndicator 1.2 by onebone for API(s) 1.6.0
LockPlayer 1.0.0 by Draemon for API(s) 1.0.0
EconomyAPI 2.0.9 by onebone for API(s) 2.0.0
AshMan 1.0.1 by FENGberd,Alcatraz_Du for API(s) 1.0.0
QituShop 1.2.0 by qitu for API(s) 1.0.0
WorldProtect 2.2.0 by aliuly for API(s) 1.10.0, 1.12.0
DontTapTheWhiteTile 0.1.2 by MUedsa for API(s) 1.8.0
SW_svile 0.6dev by svile for API(s) 2.0.0
SearchID 1.0.2 by FransicSteve for API(s) 1.0.0
InvSee 0.0.1 by tschrock (tschrock123@gmail.com) for API(s) 1.0.0
Watchdog 1.2.3 by Leo3418 for API(s) 2.0.0
EconomyLand 2.0.10 by onebone, Chinese (Simplified) translation by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
BanItem 2.2 by LDX for API(s) 1.0.0, 2.0.0
EconomyShop 2.0.7|✘wine汉化 by onebone for API(s) 1.12.0
SubnetBan 1.2.2 by for API(s) 2.0.0
SimpleAuthHelper 2.0.4 by aliuly, Chinese (Simplified) translation by Leo3418 for API(s) 1.12.0
CommandTracker 1.1 by Scott Handley for API(s) 1.10.0
RealMute 3.2.2-ch by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
MagicWE 0.2.4 by thebigsmileXD for API(s) 1.13.0, 2.0.0
EconomySell 2.0.7|✘wine汉化 by onebone for API(s) 1.12.0
DisableFlow 1.1.0 by MUedsa, upgraded by Leo3418L for API(s) 2.0.0
ColorMatch 1.1.2 by CreeperFace for API(s) 1.12.0
NameLimit 1.0.0 by Leo3418 for API(s) 2.0.0
EasyMessages 1.0.0 by Gamecrafter for API(s) 1.12.0
QD 1 by labi for API(s) 1.0.0
RestartMe 1.2.0 by Gamecrafter for API(s) 1.10.0
Prefix 1.2.0 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
FactionsPro 1.3.2 by Tethered_ for API(s) 1.12.0
essentialsTP 1.1.0 by for API(s) 1.12.0
BlockTP 1.0.3 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
KSpleef 1.3.0 by Kevin for API(s) 1.12.0
InventoryTrack 1.0.1 by Leo3418 for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
EconomyJob 2.0.4 by onebone for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.12.0, 1.13.0, 2.0.0
PocketGuard 2.1.2 by MinecrafterJPN for API(s) 1.10.0
@Leo3418 No, I didn't touch the memory limit.
Do you guys have this option on? memory.world-caches.low-memory-trigger
@Leo3418 Can you use that world editing plugin that you have and fill a huge area? If that crashes your server, do respond.
@ishitatsuyuki Yes, it is already on. @Muqsit It worked well, tried just now, no crash. However I used the plugin for cleaning up players' building in the past week, after that crash started happened frequently. Is this the root cause?
@ishitatsuyuki Yes it's on.
The cache should be destroyed if the option is on. Seems something should be wrong with it.
@Leo3418 Yes, it could be.
Thanks for helping anyways. :)
@FaanMaario Try enabling check-movement
in genisys.yml
if it is turned off:
check-movement: true
@Leo3418 Thanks :)
I confirm that is due to the memory.world-caches.low-memory-trigger
which is set to "on".
When I set it to "off", my server runs fine with no crash.
The logic seems a little backwards, but it explains the crashdump.
Hmm, low-memory-trigger
is enabled on our server, but there's no any crash since I turned on check-movement
yesterday.
The crash began when I set check-movement
to false about 10 days ago, maybe it is (also) the cause of issue.
I will do further investigation.
I also had many annoying crashes (without crashdump) with check-movement: false
. Before the server closes the CPU is eating up over 100%.
When I do "perf top" the absolute winner with the highest percentage is "zend_hash_index_find".
I needed 3 days to figure out, that the check-movement setting was the cause. I think we need some further investigations.
But because I didn't have memory problems this is probably not related with FaanMaario's issue. I'll open a new one and test without plugins etc.
Confirmed. On my server especially, when check-movement
is on, everything's OK; when it's off, server keeps crashing.
Today I turned off check-movement
again, then those familiar crash dumps reappeared.
I think it's also necessary to inspect code of the check-movement feature.
Thanks for your contributions guys, these issues will be investigated.
Mine is the opposite. When I have check-movement on 'on', the whole server takes up 100%+ CPU at times. When it is off, everything is fine.
Try disabling async chuck sending..stoped the crash for me a while ago. This might be a different issue though.
Any luck after the raklib update?
@Muqsit That wasn't a leak fix, that was a fix for a DoS bug that caused slow packet processing. It won't make any difference to this.
Is this issue still occurring? Seems tighter safety restrictions on the caches would be in order if it does.
Issue description
Since the new update of Genisys, my server constantly has these errors and crash strangely. I have removed all plugins to add them one by one to see if that was the problem but it seems not.
Steps to reproduce the issue
OS and versions
Crashdump, backtrace, memory dumps, plugins list or other files