search

iTXTech / Genisys

Feature-rich server software for Minecraft: Pocket Edition & Windows 10 Edition Beta
https://itxtech.org/genisys/
GNU General Public License v3.0
433 stars 243 forks source link

Memory issue #194

Closed Pub4Game closed 8 years ago

Pub4Game commented 8 years ago 
Genisys

 Crash Dump Sat Jan 30 12:35:10 MSK 2016

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1187840 bytes)
File: /src/pocketmine/Server
Line: 2198
Type: E_ERROR

Code:
[2189]          }
[2190]      }
[2191] 
[2192]      //$this->broadcastPacketsCallback(zlib_encode($str, ZLIB_ENCODING_DEFLATE, $this->networkCompressionLevel), $targets);//临时修复
[2193] 
[2194]      if(!$forceSync and $this->networkCompressionAsync){
[2195]          $task = new CompressBatchedTask($str, $targets, $this->networkCompressionLevel);
[2196]          $this->getScheduler()->scheduleAsyncTask($task);
[2197]      }else{
[2198]          $this->broadcastPacketsCallback(zlib_encode($str, ZLIB_ENCODING_DEFLATE, $this->networkCompressionLevel), $targets);
[2199]      }
[2200] 
[2201]      Timings::$playerNetworkTimer->stopTiming();
[2202]  }
[2203] 
[2204]  public function broadcastPacketsCallback($data, array $identifiers){
[2205]      $pk = new BatchPacket();
[2206]      $pk->payload = $data;
[2207]      $pk->encode();
[2208]      $pk->isEncoded = true;

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.0dev #0 [Protocol 38; API 2.0.0]
uname -a: Linux c1.24serv.pro 3.10.0-327.3.1.el7.x86_64 #1 SMP Wed Dec 9 14:09:15 UTC 2015 x86_64
PHP version: 7.0.2
Zend version: 3.0.0
OS : Linux, linux

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 4096 bytes)
File: /src/pocketmine/utils/Config
Line: 143
Type: E_ERROR

Code:
[134]                   case Config::PROPERTIES:
[135]                   case Config::CNF:
[136]                       $this->parseProperties($content);
[137]                       break;
[138]                   case Config::JSON:
[139]                       $this->config = json_decode($content, true);
[140]                       break;
[141]                   case Config::YAML:
[142]                       $content = self::fixYAMLIndexes($content);
[143]                       $this->config = yaml_parse($content);
[144]                       break;
[145]                   case Config::SERIALIZED:
[146]                       $this->config = unserialize($content);
[147]                       break;
[148]                   case Config::ENUM:
[149]                       $this->parseList($content);
[150]                       break;
[151]                   default:
[152]                       $this->correct = false;
[153] 

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.0dev #0 [Protocol 38; API 2.0.0]
uname -a: Linux c1.24serv.pro 3.10.0-327.3.1.el7.x86_64 #1 SMP Wed Dec 9 14:09:15 UTC 2015 x86_64
PHP version: 7.0.2
Zend version: 3.0.0
OS : Linux, linux
MrDoni98 commented 8 years ago

screenshot_2016-03-13-21-43-53 screenshot_2016-03-13-21-42-39 screenshot_2016-03-13-21-32-26 screenshot_2016-03-13-21-31-32 @PeratX the server simply stops responding

MrDoni98 commented 8 years ago

_20160313_215125 every 5 minutes

jasonczc commented 8 years ago

Caused by PTthreads.

PeratX commented 8 years ago

I do not think so. I only happens on some servers. Using XDebug now...

Muqsit commented 8 years ago

@PeratX maybe because of world ticks. I ran a timings test for 2 minutes and 80% of the lag was due to world - doTicks If a player has alot of glowstone blocks near spawn.... Ticks will increase by alot. :/ Maybe add a way to disallow block ticking?

PeratX commented 8 years ago

Seems fixed

Muqsit commented 8 years ago

@PeratX (Off-Topic) Check out beito123's FlowerPot plugin. It has fully working cauldrons and he said it's okay if we use it!

PeratX commented 8 years ago

@Muqsit Link please

Muqsit commented 8 years ago

@PeratX sorry for the late reply. https://github.com/beito123/PocketMine-MP-Plugins/tree/master/test/FlowerPot

ghost commented 8 years ago

I did not repaired even put . Also do you have relationship pocketmine.yml?

PeratX commented 8 years ago

Yes. But we can not trace ...

XxPeaceHellxX commented 8 years ago

Fix Crash Please

ghost commented 8 years ago

fix please. 2016-03-26 [00:45:20] [Server thread/INFO]: Mark_Zenenko_YT has just earned the achievement Time to Mine! 2016-03-26 [00:45:46] [Server thread/EMERGENCY]: An unrecoverable error has occurred and the server has crashed. Creating a crash dump 2016-03-26 [00:45:46] [Server thread/EMERGENCY]: Please upload the "/root/serv/crashdumps/CrashDump_Sat_Mar_26-00.45.46-CDT_2016.log" file to the Crash Archive and submit the link to the Bug Reporting page. Give as much info as you can.

Genisys Crash Dump Sat Mar 26 00:45:46 CDT 2016

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 8192 bytes)
File: /src/pocketmine/utils/Utils
Line: 233
Type: E_ERROR

Code:
[224]       return self::$os;
[225]   }
[226] 
[227] 
[228]   public static function getRealMemoryUsage(){
[229]       $stack = 0;
[230]       $heap = 0;
[231] 
[232]       if(Utils::getOS() === "linux" or Utils::getOS() === "android"){
[233]           $mappings = file("/proc/self/maps");
[234]           foreach($mappings as $line){
[235]               if(preg_match("#([a-z0-9]+)\\-([a-z0-9]+) [rwxp\\-]{4} [a-z0-9]+ [^\\[]*\\[([a-zA-z0-9]+)\\]#", trim($line), $matches) > 0){
[236]                   if(strpos($matches[3], "heap") === 0){
[237]                       $heap += hexdec($matches[2]) - hexdec($matches[1]);
[238]                   }elseif(strpos($matches[3], "stack") === 0){
[239]                       $stack += hexdec($matches[2]) - hexdec($matches[1]);
[240]                   }
[241]               }
[242]           }
[243]       }

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.1dev #0 [Protocol 45; API 2.0.0]
uname -a: Linux domain 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64
PHP version: 7.0.2
Zend version: 3.0.0
OS : Linux, linux
xpyctum commented 8 years ago

Учимся записывать Краши в блок "код". " ''' код с крашем ''' "

ishitatsuyuki commented 8 years ago

No "fix please". It's a hard problem. Any code helps are welcome.

NML32 commented 8 years ago

anyone else still having memory issues? I am and I wish more effort was put in to fixing it.

gaoyichuan commented 8 years ago

I haven't get it since last week, but I don't know whether it's because I have no player

udachin26 commented 8 years ago

When a bug fix?

beetree commented 8 years ago

This is a pthreads issue.

beetree commented 8 years ago

I just got this issue again:

Genisys Crash Dump Tue Apr 26 03:24:05 UTC 2016

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 86016 bytes)
File: /src/pocketmine/Player
Line: 4599
Type: E_ERROR

Code:
[4590]  public static function getChunkCacheFromData($chunkX, $chunkZ, $payload, $ordering = FullChunkDataPacket::ORDER_COLUMNS){
[4591]          $pk = new FullChunkDataPacket();
[4592]          $pk->chunkX = $chunkX;
[4593]          $pk->chunkZ = $chunkZ;
[4594]          $pk->order = $ordering;
[4595]          $pk->data = $payload;
[4596]          if(Network::$BATCH_THRESHOLD >= 0){
[4597]                  $pk->encode();
[4598]                  $batch = new BatchPacket();
[4599]                  $batch->payload = zlib_encode(Binary::writeInt(strlen($pk->getBuffer())) . $pk->getBuffer(), ZLIB_ENCODING_DEFLATE, Server::getInstance()->networkCompressionLevel);
[4600]                  $batch->encode();
[4601]                  $batch->isEncoded = true;
[4602]                  return $batch;
[4603]          }
[4604]          return $pk;
[4605]  }
[4606]
[4607]
[4608] }
[4609]

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.1dev #0 [Protocol 46; API 2.0.0]
uname -a: Linux m3033.contabo.host 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64
PHP version: 7.0.3
Zend version: 3.0.0
OS : Linux, linux

Loaded plugins:
加载的插件:
Slapper 1.2.9.3 by jojoe77777 for API(s) 1.12.0, 2.0.0
FactionsPro 1.3.2lm4 by Tethered_ for API(s) 1.12.0, 2.0.0
PurePerms 1.3.6 by 64FF00 for API(s) 1.13.0, 2.0.0
TapToDo-LEET.CC 1.1.0 by Falk, 64FF00 for API(s) 2.0.0
BlockedCommands 1.1.0 by ProjectInfinity for API(s) 2.0.0
LeetAuth 1.3.2 by ProjectInfinity for API(s) 2.0.0
BanItem 2.1 by LDX for API(s) 2.0.0
LeetTP 1.1.0 by ProjectInfinity for API(s) 2.0.0
Nick 1.0.0 by ProjectInfinity for API(s) 1.12.0, 2.0.0
SignStatus 3.0.0 by xpyctum for API(s) 2.0.0
ClearLagg 1.1.0 by LegendOfMCPE for API(s) 2.0.0
Economy2 0.1.1 by ProjectInfinity for API(s) 2.0.0
NoAdvertisingPE 1.0.0 by Vaivez66 for API(s) 2.0.0
DevTools 1.11.0 by PocketMine Team for API(s) 2.0.0
Lightning_Strike 2 by MichaelM04 for API(s) 2.0.0
ServerLoveMCPE 3.3 by TheDeibo, ratchetgame98, and, The Plugin Community! for API(s) 2.0.0
Rainbow_Effect 1.0.0 by LilCrispy2o9/Angelo Vidrio for API(s) 2.0.0
EconomyAPI 0.1.0 by ProjectInfinity for API(s) 2.0.0
LeetLogin 1.0.0 by ProjectInfinity for API(s) 2.0.0
AdminFun 1.0.0 by hoyinm14mc for API(s) 1.12.0, 1.13.0, 2.0.0
MineReset 2.2 by Falk for API(s) 2.0.0
Alias 1.1.2m by ZacHack, amended by madhon for API(s) 1.1.0, 2.0.0
WorldEditArt 1.0.0_beta_dev_#052-leet by Array for API(s) 2.0.0
PeacefulSpawn 2.5.1 by LDX for API(s) 1.0.0, 2.0.0
ItemCasePE 1.0.8 by aliuly for API(s) 2.0.0
iProtector 3.1bm2 by LDX for API(s) 1.9.0, 2.0.0
DevBAN 1 by Ragnok123 for API(s) 2.0.0
ChatFilter 1.1 by onebone for API(s) 2.0.0
LeetInfoViewer 1.0.0 by ProjectInfinity for API(s) 2.0.0
EconomyJob 2.0.4 by onebone for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 2.0.0
EconomyCasino 2.0.2 by onebone for API(s) 2.0.0
EconomyAirport 1337.0.0 by onebone & ProjectInfinity for API(s) 2.0.0
EconomyTax 2.0.3 by onebone for API(s) 2.0.0
Economy2Shop 1.0.0 by ProjectInfinity for API(s) 2.0.0
EconomyAuction 2.0.2 by onebone for API(s) 2.0.0
PureChat 1.4.0.2 by 64FF00 for API(s) 1.13.0, 2.0.0
ClockProgramming commented 8 years ago

Maybe this is a memory leak

ghost commented 8 years ago

I have the same thing. when the server is on, i notice as time goes by, it uses more, and more ram until my pc ram is done, then the server shuts down.

Muqsit commented 8 years ago

Post your pocketmine.yml file

ghost commented 8 years ago

rename it to .yml since it wont let me post .zip or .yml pocketmineyml_renameto.yml.txt

kaitododoyt commented 8 years ago

turn memory limit into 0 :P

beetree commented 8 years ago

Still happening:

[leet@m2600 crashdumps]$ cat CrashDump_Sun_May_1-19.44.12-UTC_2016.log
Genisys Crash Dump Sun May 1 19:44:12 UTC 2016

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1207959552 bytes)
File: /src/pocketmine/Server
Line: 1603
Type: E_ERROR

Code:
[1594]                  Server::$sleeper->wait($ms);
[1595]          }, $microseconds);
[1596]  }
[1597]
[1598]  public function getExpectedExperience($level){
[1599]          if(isset($this->expCache[$level])) return $this->expCache[$level];
[1600]          $levelSquared = $level ** 2;
[1601]          if($level < 16) $this->expCache[$level] = $levelSquared + 6 * $level;
[1602]          elseif($level < 31) $this->expCache[$level] = 2.5 * $levelSquared - 40.5 * $level + 360;
[1603]          else $this->expCache[$level] = 4.5 * $levelSquared - 162.5 * $level + 2220;
[1604]          return $this->expCache[$level];
[1605]  }
[1606]
[1607]  public function about(){
[1608]          $this->logger->info($this->aboutstring);
[1609]  }
[1610]
[1611]  public function loadAdvancedConfig(){
[1612]          $this->playerMsgType = $this->getAdvancedProperty("server.player-msg-type", self::PLAYER_MSG_TYPE_MESSAGE);
[1613]          $this->playerLoginMsg = $this->getAdvancedProperty("server.login-msg", "§3@player joined the game");

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.1dev #0 [Protocol 60; API 2.0.0]
uname -a: Linux m2600.contabo.host 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64
PHP version: 7.0.3
Zend version: 3.0.0
OS : Linux, linux

Loaded plugins:
加载的插件:
NoAdvertisingPE 1.0.0 by Vaivez66 for API(s) 2.0.0
TapToDo-LEET.CC 1.1.0 by Falk, 64FF00 for API(s) 2.0.0
ClearLagg 1.1.0 by LegendOfMCPE for API(s) 2.0.0
FactionsPro 1.3.2lm4 by Tethered_ for API(s) 1.12.0, 2.0.0
PurePerms 1.3.6 by 64FF00 for API(s) 1.13.0, 2.0.0
BlockedCommands 1.1.0 by ProjectInfinity for API(s) 2.0.0
LeetTP 1.1.0 by ProjectInfinity for API(s) 2.0.0
MoreCommands 1.1.4 by  for API(s) 2.0.0
SignStatus 3.0.0 by xpyctum for API(s) 2.0.0
LeetAuth 1.3.2 by ProjectInfinity for API(s) 2.0.0
TimerBan 2.2 by onebone for API(s) 2.0.0
AdminFun 1.0.0 by hoyinm14mc for API(s) 1.12.0, 1.13.0, 2.0.0
Economy2 0.1.1 by ProjectInfinity for API(s) 2.0.0
Slapper 1.2.9.3 by jojoe77777 for API(s) 1.12.0, 2.0.0
HealthStats 1.1.1 by Striker209_MPE for API(s) 2.0.0
DevTools 1.11.0 by PocketMine Team for API(s) 2.0.0
Lightning_Strike 2 by MichaelM04 for API(s) 2.0.0
1vs1 0.0.4 by  for API(s) 2.0.0
Rainbow_Effect 1.0.0 by LilCrispy2o9/Angelo Vidrio for API(s) 2.0.0
LeetLogin 1.0.0 by ProjectInfinity for API(s) 2.0.0
Alias 1.1.2m by ZacHack, amended by madhon for API(s) 1.1.0, 2.0.0
WorldEditArt 1.0.0_beta_dev_#052-leet by Array for API(s) 2.0.0
PeacefulSpawn 2.5.1 by LDX for API(s) 1.0.0, 2.0.0
ItemCasePE 1.0.8 by aliuly for API(s) 2.0.0
iProtector 3.1bm2 by LDX for API(s) 1.9.0, 2.0.0
ChatFilter 1.1 by onebone for API(s) 2.0.0
EconomyAirport 1337.0.0 by onebone & ProjectInfinity for API(s) 2.0.0
Economy2Shop 1.0.0 by ProjectInfinity for API(s) 2.0.0
LeetInfoViewer 1.0.0 by ProjectInfinity for API(s) 2.0.0
EconomyAPI 0.1.0 by ProjectInfinity for API(s) 2.0.0
EconomyJob 2.0.4 by onebone for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 2.0.0
EconomyAuction 2.0.2 by onebone for API(s) 2.0.0
EconomyCasino 2.0.2 by onebone for API(s) 2.0.0
EconomyTax 2.0.3 by onebone for API(s) 2.0.0
PureChat 1.4.0.3 by 64FF00 for API(s) 1.13.0, 2.0.0

@Muqsit below is pocketmine.yml:

# Main configuration file for PocketMine-MP
# These settings are the ones that cannot be included in server.properties
# Some of these settings are safe, others can break your server if modified incorrectly
# New settings/defaults won't appear automatically on this file when upgrading.

settings:
 #Three-letter language code for server-side localization
 #Check your language code on https://en.wikipedia.org/wiki/List_of_ISO_639-2_codes
 language: "eng"
 #Whether to send all strings translated to server locale or let the device handle them
 force-language: false
 #When server shut down, the players will get kicked and this is what will show on there screen.
 shutdown-message: "Server closed"
 #Allow listing plugins via Query
 query-plugins: true
 #Show a console message when a plugin uses deprecated API methods
 deprecated-verbose: true
 #Enable plugin and core profiling by default
 enable-profiling: false
 #Will only add results when tick measurement is below or equal to given value (default 20)
 profile-report-trigger: 20
 #Number of AsyncTask workers.
 #Used for plugin asynchronous tasks, world generation, compression and web communication.
 #Set this approximately to your number of cores.
 #If set to auto, it'll try to detect the number of cores (or use 2)
 async-workers: auto

memory:
 #Global soft memory limit in megabytes. Set to 0 to disable
 #This will trigger low-memory-triggers and fire an event to free memory when the usage goes over this
 global-limit: 0

 #Main thread soft memory limit in megabytes. Set to 0 to disable
 #This will trigger low-memory-triggers and fire an event to free memory when the usage goes over this
 main-limit: 0

 #Main thread hard memory limit in megabytes. Set to 0 to disable
 #This will stop the server when the limit is surpassed
 main-hard-limit: 1024

 #Period in ticks to check memory (default 1 second)
 check-rate: 20

 #Continue firing low-memory-triggers and event while on low memory
 continuous-trigger: true

 #Only if memory.continuous-trigger is enabled. Specifies the rate in memory.check-rate steps (default 30 seconds)
 continuous-trigger-rate: 30

 garbage-collection:
  #Period in ticks to fire the garbage collector manually (default 30 minutes), set to 0 to disable
  #This only affect the main thread. Other threads should fire their own collections
  period: 36000

  #Fire asynchronous tasks to collect garbage from workers
  collect-async-worker: true

  #Trigger on low memory
  low-memory-trigger: true

 max-chunks:
  #Limit of chunks to load per player, overrides chunk-sending.max-chunks
  trigger-limit: 28

  #Do chunk garbage collection on trigger
  trigger-chunk-collect: true

  #Trigger on low memory
  low-memory-trigger: true

 world-caches:
  disable-chunk-cache: true
  low-memory-trigger: true

network:
 #Threshold for batching packets, in bytes. Only these packets will be compressed
 #Set to 0 to compress everything, -1 to disable.
 batch-threshold: 256
 #Compression level used when sending batched packets. Higher = Uses More CPU, Less = More Bandwidth Usage
 compression-level: 2
 #Use AsyncTasks for compression. Adds half/one tick delay, less CPU load on main thread
 async-compression: true
 #Experimental, only for Windows. Tries to use UPnP to automatically port forward
 upnp-forwarding: false

debug:
 #If > 1, it will show debug messages in the console
 level: 1
 #Enables /status, /gc
 commands: true

level-settings:
 #The default format that levels will use when created
 default-format: mcregion
 #If true, converts from a format that is not the default to the default format on load
 #NOTE: This is currently not implemented
 convert-format: false
 #Automatically change levels tick rate to maintain 20 ticks per second
 auto-tick-rate: false
 auto-tick-rate-limit: 20
 #Sets the base tick rate (1 = 20 ticks per second, 2 = 10 ticks per second, etc.)
 base-tick-rate: 1
 #Tick all players each tick even when other settings disallow this.
 always-tick-players: false

chunk-sending:
 #Amount of chunks sent to players per tick
 per-tick: 1
 #Amount of chunks sent around each player
 max-chunks: 32
 #Amount of chunks that need to be sent before spawning the player
 spawn-threshold: 28
 #Save a serialized copy of the chunk in memory for faster sending
 #Useful in mostly-static worlds where lots of players join at the same time
 cache-chunks: true

chunk-ticking:
 #Max amount of chunks processed each tick
 per-tick: 0
 #Radius of chunks around a player to tick
 tick-radius: 0
 light-updates: false
 clear-tick-list: true

chunk-generation:
 #Max. amount of chunks in the waiting queue to be generated
 queue-size: 2
 #Max. amount of chunks in the waiting queue to be populated
 population-queue-size: 2

ticks-per:
 animal-spawns: 400
 monster-spawns: 1
 autosave: 6000
 cache-cleanup: 900
spawn-limits:
  monsters: 70
  animals: 15
  water-animals: 5
  ambient: 15

auto-report:
 #Send crash reports for processing
 enabled: false
 send-code: true
 send-settings: true
 send-phpinfo: false
 host: crash.pocketmine.net

anonymous-statistics:
 #Sends anonymous statistics for data aggregation, plugin usage tracking
 enabled: true
 host: stats.pocketmine.net

auto-updater:
 enabled: false
 on-update:
  warn-console: false
  warn-ops: false
 #Can be development, beta or stable.
 preferred-channel: beta
 #If using a development version, it will suggest changing the channel
 suggest-channels: false
 host: www.pocketmine.net

aliases:
 #Examples:
 #showtheversion: version
 #savestop: [save-all, stop]

worlds:
 #These settings will override the generator set in server.properties and allows loading multiple levels
 #Example:
 #world:
 # seed: 404
 # generator: FLAT:2;7,59x1,3x3,2;1;decoration(treecount=80 grasscount=45)
MaryVictoria commented 8 years ago

this happen to me when i worldedit larger area the server closes then out of memory. On May 5, 2016 9:39 PM, "beetree" notifications@github.com wrote:

Still happening:

[leet@m2600 crashdumps]$ cat CrashDump_Sun_May_1-19.44.12-UTC_2016.log Genisys Crash Dump Sun May 1 19:44:12 UTC 2016

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1207959552 bytes) File: /src/pocketmine/Server Line: 1603 Type: E_ERROR

Code: [1594] Server::$sleeper->wait($ms); [1595] }, $microseconds); [1596] } [1597] [1598] public function getExpectedExperience($level){ [1599] if(isset($this->expCache[$level])) return $this->expCache[$level]; [1600] $levelSquared = $level * 2; [1601] if($level < 16) $this->expCache[$level] = $levelSquared + 6 * $level; [1602] elseif($level < 31) $this->expCache[$level] = 2.5 * $levelSquared - 40.5 * $level + 360; [1603] else $this->expCache[$level] = 4.5 * $levelSquared - 162.5 \ $level + 2220; [1604] return $this->expCache[$level]; [1605] } [1606] [1607] public function about(){ [1608] $this->logger->info($this->aboutstring); [1609] } [1610] [1611] public function loadAdvancedConfig(){ [1612] $this->playerMsgType = $this->getAdvancedProperty("server.player-msg-type", self::PLAYER_MSG_TYPE_MESSAGE); [1613] $this->playerLoginMsg = $this->getAdvancedProperty("server.login-msg", "§3@player joined the game");

Backtrace:

0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.1dev #0 [Protocol 60; API 2.0.0] uname -a: Linux m2600.contabo.host 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 PHP version: 7.0.3 Zend version: 3.0.0 OS : Linux, linux

Loaded plugins: 加载的插件: NoAdvertisingPE 1.0.0 by Vaivez66 for API(s) 2.0.0 TapToDo-LEET.CC 1.1.0 by Falk, 64FF00 for API(s) 2.0.0 ClearLagg 1.1.0 by LegendOfMCPE for API(s) 2.0.0 FactionsPro 1.3.2lm4 by Tethered_ for API(s) 1.12.0, 2.0.0 PurePerms 1.3.6 by 64FF00 for API(s) 1.13.0, 2.0.0 BlockedCommands 1.1.0 by ProjectInfinity for API(s) 2.0.0 LeetTP 1.1.0 by ProjectInfinity for API(s) 2.0.0 MoreCommands 1.1.4 by for API(s) 2.0.0 SignStatus 3.0.0 by xpyctum for API(s) 2.0.0 LeetAuth 1.3.2 by ProjectInfinity for API(s) 2.0.0 TimerBan 2.2 by onebone for API(s) 2.0.0 AdminFun 1.0.0 by hoyinm14mc for API(s) 1.12.0, 1.13.0, 2.0.0 Economy2 0.1.1 by ProjectInfinity for API(s) 2.0.0 Slapper 1.2.9.3 by jojoe77777 for API(s) 1.12.0, 2.0.0 HealthStats 1.1.1 by Striker209_MPE for API(s) 2.0.0 DevTools 1.11.0 by PocketMine Team for API(s) 2.0.0 Lightning_Strike 2 by MichaelM04 for API(s) 2.0.0 1vs1 0.0.4 by for API(s) 2.0.0 Rainbow_Effect 1.0.0 by LilCrispy2o9/Angelo Vidrio for API(s) 2.0.0 LeetLogin 1.0.0 by ProjectInfinity for API(s) 2.0.0 Alias 1.1.2m by ZacHack, amended by madhon for API(s) 1.1.0, 2.0.0 WorldEditArt 1.0.0_betadev#052-leet by Array for API(s) 2.0.0 PeacefulSpawn 2.5.1 by LDX for API(s) 1.0.0, 2.0.0 ItemCasePE 1.0.8 by aliuly for API(s) 2.0.0 iProtector 3.1bm2 by LDX for API(s) 1.9.0, 2.0.0 ChatFilter 1.1 by onebone for API(s) 2.0.0 EconomyAirport 1337.0.0 by onebone & ProjectInfinity for API(s) 2.0.0 Economy2Shop 1.0.0 by ProjectInfinity for API(s) 2.0.0 LeetInfoViewer 1.0.0 by ProjectInfinity for API(s) 2.0.0 EconomyAPI 0.1.0 by ProjectInfinity for API(s) 2.0.0 EconomyJob 2.0.4 by onebone for API(s) 1.0.0, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 2.0.0 EconomyAuction 2.0.2 by onebone for API(s) 2.0.0 EconomyCasino 2.0.2 by onebone for API(s) 2.0.0 EconomyTax 2.0.3 by onebone for API(s) 2.0.0 PureChat 1.4.0.3 by 64FF00 for API(s) 1.13.0, 2.0.0

@Muqsit https://github.com/Muqsit below is pocketmine.yml:

Main configuration file for PocketMine-MP

These settings are the ones that cannot be included in server.properties

Some of these settings are safe, others can break your server if modified incorrectly

New settings/defaults won't appear automatically on this file when upgrading.

settings:

Three-letter language code for server-side localization

Check your language code on https://en.wikipedia.org/wiki/List_of_ISO_639-2_codes

language: "eng"

Whether to send all strings translated to server locale or let the device handle them

force-language: false

When server shut down, the players will get kicked and this is what will show on there screen.

shutdown-message: "Server closed"

Allow listing plugins via Query

query-plugins: true

Show a console message when a plugin uses deprecated API methods

deprecated-verbose: true

Enable plugin and core profiling by default

enable-profiling: false

Will only add results when tick measurement is below or equal to given value (default 20)

profile-report-trigger: 20

Number of AsyncTask workers.

Used for plugin asynchronous tasks, world generation, compression and web communication.

Set this approximately to your number of cores.

If set to auto, it'll try to detect the number of cores (or use 2)

async-workers: auto

memory:

Global soft memory limit in megabytes. Set to 0 to disable

This will trigger low-memory-triggers and fire an event to free memory when the usage goes over this

global-limit: 0

Main thread soft memory limit in megabytes. Set to 0 to disable

This will trigger low-memory-triggers and fire an event to free memory when the usage goes over this

main-limit: 0

Main thread hard memory limit in megabytes. Set to 0 to disable

This will stop the server when the limit is surpassed

main-hard-limit: 1024

Period in ticks to check memory (default 1 second)

check-rate: 20

Continue firing low-memory-triggers and event while on low memory

continuous-trigger: true

Only if memory.continuous-trigger is enabled. Specifies the rate in memory.check-rate steps (default 30 seconds)

continuous-trigger-rate: 30

garbage-collection:

Period in ticks to fire the garbage collector manually (default 30 minutes), set to 0 to disable

This only affect the main thread. Other threads should fire their own collections

period: 36000

Fire asynchronous tasks to collect garbage from workers

collect-async-worker: true

Trigger on low memory

low-memory-trigger: true

max-chunks:

Limit of chunks to load per player, overrides chunk-sending.max-chunks

trigger-limit: 28

Do chunk garbage collection on trigger

trigger-chunk-collect: true

Trigger on low memory

low-memory-trigger: true

world-caches: disable-chunk-cache: true low-memory-trigger: true

network:

Threshold for batching packets, in bytes. Only these packets will be compressed

Set to 0 to compress everything, -1 to disable.

batch-threshold: 256

Compression level used when sending batched packets. Higher = Uses More CPU, Less = More Bandwidth Usage

compression-level: 2

Use AsyncTasks for compression. Adds half/one tick delay, less CPU load on main thread

async-compression: true

Experimental, only for Windows. Tries to use UPnP to automatically port forward

upnp-forwarding: false

debug:

If > 1, it will show debug messages in the console

level: 1

Enables /status, /gc

commands: true

level-settings:

The default format that levels will use when created

default-format: mcregion

If true, converts from a format that is not the default to the default format on load

NOTE: This is currently not implemented

convert-format: false

Automatically change levels tick rate to maintain 20 ticks per second

auto-tick-rate: false auto-tick-rate-limit: 20

Sets the base tick rate (1 = 20 ticks per second, 2 = 10 ticks per second, etc.)

base-tick-rate: 1

Tick all players each tick even when other settings disallow this.

always-tick-players: false

chunk-sending:

Amount of chunks sent to players per tick

per-tick: 1

Amount of chunks sent around each player

max-chunks: 32

Amount of chunks that need to be sent before spawning the player

spawn-threshold: 28

Save a serialized copy of the chunk in memory for faster sending

Useful in mostly-static worlds where lots of players join at the same time

cache-chunks: true

chunk-ticking:

Max amount of chunks processed each tick

per-tick: 0

Radius of chunks around a player to tick

tick-radius: 0 light-updates: false clear-tick-list: true

chunk-generation:

Max. amount of chunks in the waiting queue to be generated

queue-size: 2

Max. amount of chunks in the waiting queue to be populated

population-queue-size: 2

ticks-per: animal-spawns: 400 monster-spawns: 1 autosave: 6000 cache-cleanup: 900 spawn-limits: monsters: 70 animals: 15 water-animals: 5 ambient: 15

auto-report:

Send crash reports for processing

enabled: false send-code: true send-settings: true send-phpinfo: false host: crash.pocketmine.net

anonymous-statistics:

Sends anonymous statistics for data aggregation, plugin usage tracking

enabled: true host: stats.pocketmine.net

auto-updater: enabled: false on-update: warn-console: false warn-ops: false

Can be development, beta or stable.

preferred-channel: beta

If using a development version, it will suggest changing the channel

suggest-channels: false host: www.pocketmine.net

aliases:

Examples:

showtheversion: version

savestop: [save-all, stop]

worlds:

These settings will override the generator set in server.properties and allows loading multiple levels

Example:

world:

seed: 404

generator: FLAT:2;7,59x1,3x3,2;1;decoration(treecount=80 grasscount=45)

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/iTXTech/Genisys/issues/194#issuecomment-217156814

ghost commented 8 years ago

Guys the server was running normally then this happens

26.05 18:51:13 [Server] INFO Warning: include(phar:///jar/versions/Genisys/Genisys405.phar/src/pocketmine/entity/FallingSand.php): failed to open stream: phar error: internal corruption of phar "/jar/versions/Genisys/Genisys405.phar" (actual filesize mismatch on file "src/pocketmine/entity/FallingSand.php") in phar:///jar/versions/Genisys/Genisys405.phar/src/spl/BaseClassLoader.php on line 128 26.05 18:51:13 [Server] INFO Warning: include(): Failed opening 'phar:///jar/versions/Genisys/Genisys405.phar/src/pocketmine/entity/FallingSand.php' for inclusion (include_path='.:') in phar:///jar/versions/Genisys/Genisys405.phar/src/spl/BaseClassLoader.php on line 128 26.05 18:51:13 [Server] INFO Warning: include(phar:///jar/versions/Genisys/Genisys405.phar/src/spl/ClassNotFoundException.php): failed to open stream: phar error: internal corruption of phar "/jar/versions/Genisys/Genisys405.phar" (actual filesize mismatch on file "src/spl/ClassNotFoundException.php") in phar:///jar/versions/Genisys/Genisys405.phar/src/spl/BaseClassLoader.php on line 128 26.05 18:51:13 [Server] INFO Warning: include(): Failed opening 'phar:///jar/versions/Genisys/Genisys405.phar/src/spl/ClassNotFoundException.php' for inclusion (include_path='.:') in phar:///jar/versions/Genisys/Genisys405.phar/src/spl/BaseClassLoader.php on line 128 26.05 18:51:13 [Server] Server thread/CRITICAL Error: "Class 'ClassNotFoundException' not found" (EXCEPTION) in "/src/spl/BaseClassLoader" at line 131 26.05 18:51:13 [Server] Server thread/EMERGENCY An unrecoverable error has occurred and the server has crashed. Creating a crash dump 26.05 18:51:13 [Multicraft] Server shut down (starting) 26.05 18:51:13 [Multicraft] Restarting crashed server in 5 seconds 26.05 18:51:13 [Multicraft] Stopping server! 26.05 18:51:19 [Multicraft] Received stop command

ishitatsuyuki commented 8 years ago

@mote166 unrelated issue. Didn't you see 'internal phar corruption'?

Muqsit commented 8 years ago

According to PHP7.0.7's changelog, this issue is fixed. Idk if im right...

Description:

A memory leak occurs when a closure that is defined inside a class and has a parameter named $this is invoked with a value that contains a reference to $this.

This gist (https://gist.github.com/jmalloc/e3db5842c2c4ab2a1edf) might help explain a little better.

Affected PHP versions: https://3v4l.org/lr963

Test script:

class MemoryLeak { public function bad() { $closure = function ($this) {}; $closure([$this]); } }

$object = new MemoryLeak; ini_set('memory_limit', '10M');

for ($i = 0; $i < 1000000; ++$i) { $object->bad(); }

echo "Done" . PHP_EOL;

Expected result:

Done

Actual result:

Fatal error: Allowed memory size of 10485760 bytes exhausted (tried to allocate 4096 bytes) in /Users/james//leak.php on line 8

ishitatsuyuki commented 8 years ago

@Muqsit AFAIK I don't think we use closure. It's either a pthreads issue or a cyclic reference.

Muqsit commented 8 years ago

Oh! I am a newbie to PHP.

fihzy commented 8 years ago

I am having a bad memory leak issue. I am using php 7.0.7. I compiled it myself using these options:

./configure --with-tsrm-pthreads --enable-maintainer-zts --enable-phar --enable-sockets --with-zlib --with-curl --enable-bcmath --enable-mbstring=all --enable-calendar --enable-phar --enable-ctype --enable-shmp --disable-short-tags --enable-cli --enable-zip --enable-ftp --enable-opcache --enable-weakref --enable-bcmath --with-pic --disable-cgi --disable-session --disable-pdo --disable-xmlreader --disable-xmlwriter --disable-dom --with-gd --with-png --with-yaml --with-gmp --with-mcrypt --with-zlib --with-curl --with-png

When I fire up the server, if no clients connect, the memory usage is stable. If just a single client connects, spends a minute or two on the server and then immediately leaves, and NO MORE clients join, the memory usage of the process continues to grow and grow until it exhausts the machine. Eventually the process gets jammed at 100% cpu and starts showing "overloaded?" messages. I am not sure if the CPU spike is a separate issue but I haven't seen it happen other than when memory usage is extraordinary.

The attached file is a log of memory usage of php during a 16 hour run. No clients were on the server for the entire duration after the 1st client disconnected right away. The file has the format in time,vsz,rss memory usage.

memory.txt

Pub4Game commented 8 years ago 
Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 335872 bytes)
File: /src/pocketmine/Server
Line: 2272
Type: E_ERROR

Code:
[2263]  public function batchPackets(array $players, array $packets, $forceSync = false){
[2264]      Timings::$playerNetworkTimer->startTiming();
[2265]      $str = "";
[2266] 
[2267]      foreach($packets as $p){
[2268]          if($p instanceof DataPacket){
[2269]              if(!$p->isEncoded){
[2270]                  $p->encode();
[2271]              }
[2272]              $str .= Binary::writeInt(strlen($p->buffer)) . $p->buffer;
[2273]          }else{
[2274]              $str .= Binary::writeInt(strlen($p)) . $p;
[2275]          }
[2276]      }
[2277] 
[2278]      $targets = [];
[2279]      foreach($players as $p){
[2280]          if($p->isConnected()){
[2281]              $targets[] = $this->identifiers[spl_object_hash($p)];
[2282]          }

Backtrace:
#0 (): pocketmine\Server->crashDump(boolean)

Genisys version: 1.1dev #416 [Protocol 70; API 2.0.0]
uname -a: Linux s1.24serv.pro 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64
PHP version: 7.0.6
Zend version: 3.0.0
OS : Linux, linux

@PeratX

dterweij commented 8 years ago

A no go yet for me to move to Genisys. gs screenshot_1

NML32 commented 8 years ago

I think I'm going to have to stop using Genisys since all of my servers are crashing with the out of memory issue. This really sucks.

Muqsit commented 8 years ago

Do you know that from the top 25 servers, 20 of them run on Genisys and guess what? They do not experience out of memory issues. Don't blame the software all the time. You can try optimizing Genisys through Genisys.yml, pocketmine.yml and server.properties file. Also, loading large number of worlds and plugins is bad because it will just eat up the memory. So make sure you're using minimal features. Again, that's PHP, that's why you need to optimize the software to its best.

Muqsit commented 8 years ago

This issue is interesting, its something unique. But I am just saying, you could do that ^ rather than waiting for a fix from us. Try decreasing the max chunks to 26 (default is 96 which is a lot)

fihzy commented 8 years ago

So for my issue, as commented on earlier,I have 1 world, no plugins and nothing fancy at all. This issue still happens. If the server is fired up fresh, and no clients connect, it sits there using state amounts of RAM. The minute one user connects, even if they disconnect seconds later, and no other clients join, the server starts immediately consuming extra ram all the way to the point where it uses whatever allocation it has.

I even tried using a PHP compiled with the provided build scripts, rather than my own versions, and that changed nothing either.

fihzy commented 8 years ago

I thought it was something to do with my particular world, maybe a corruption in the file or some attribute of the world causing this to occur, but it wasn't. I start a brand new world with no placed blocks other than what was created from seed, and it STILL happened.

NML32 commented 8 years ago

@lastrolo I have the same issue on all my servers running Genisys and I've tried the above suggestions and I no improvements. I'm trying to hold out as long as I can in hopes they find a solution to the memory issues.

ClockProgramming commented 8 years ago

Never had this memory issue.... Even though I didn't changed anything yet from my pocketmine.yml and genisys.yml On Jun 12, 2016 10:19 PM, "NML32" notifications@github.com wrote:

@lastrolo https://github.com/lastrolo I have the same issue on all my servers running Genisys and I've tried the above suggestions and I no improvements. I'm trying to hold out as long as I can in hopes they find a solution to the memory issues.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/iTXTech/Genisys/issues/194#issuecomment-225437557, or mute the thread https://github.com/notifications/unsubscribe/APv_xQCqNW-rqaFYejfWTkmLKZo-Zhubks5qLBVWgaJpZM4HPxK3 .

ClockProgramming commented 8 years ago

Perhaps computer problem ๑乛◡乛๑ On Jun 13, 2016 9:10 AM, "Vincent Paul Tampos" glitchplayer21@gmail.com wrote:

Never had this memory issue.... Even though I didn't changed anything yet from my pocketmine.yml and genisys.yml On Jun 12, 2016 10:19 PM, "NML32" notifications@github.com wrote:

@lastrolo https://github.com/lastrolo I have the same issue on all my servers running Genisys and I've tried the above suggestions and I no improvements. I'm trying to hold out as long as I can in hopes they find a solution to the memory issues.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/iTXTech/Genisys/issues/194#issuecomment-225437557, or mute the thread https://github.com/notifications/unsubscribe/APv_xQCqNW-rqaFYejfWTkmLKZo-Zhubks5qLBVWgaJpZM4HPxK3 .

fihzy commented 8 years ago

DO we have anything in common, people who are having issues? My server is running on a VPS host (openvz). 4 cpu, 3gb of ram dedicated. The kernel is only a 2.6.32 kernel it looks like.

dktapps commented 8 years ago 
Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 835584 bytes)
File: /src/pocketmine/Server
Line: 2246
Type: E_ERROR

and this:

Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 131072 bytes)
File: /src/pocketmine/nbt/NBT
Line: 499
Type: E_ERROR

I see these crashdumps on my friend's server daily. It's always one of the above two errors.

dktapps commented 8 years ago

I'm frequently seeing zlib_encode on the lines pointed out by the crashdumps.

ishitatsuyuki commented 8 years ago

@dktapps this is due to it allocates a lot of memory and likely to trigger memory limit.

fihzy commented 8 years ago

I mention this will all due respect and only to try and further the discussion; ClearSky works without leaking memory, using the exact same PHP binary on the exact same VPS. So it really does seem be something code related and not "environmental" in terms of the setup the code executes in.

dktapps commented 8 years ago

Would people experiencing leaks like to try a phar from the memory-leak-fix branch? https://gitlab.com/itxtech/genisys/builds/1983031/artifacts/download

If you choose to use this, please be aware that this is an experimental build and there may be issues with it of which I am currently unaware.

fihzy commented 8 years ago

Interesting. Can you elaborate on what has changed with this version? It appears to be running stable on my box now. Here is the memory log from start up, and the last line has been maintained now for 8 hours. I have never seen that sort of stability.

Thu Jun 23 23:24:16 CDT 2016 601872 91248 Thu Jun 23 23:24:21 CDT 2016 601872 91248 Thu Jun 23 23:24:26 CDT 2016 601872 91248 Thu Jun 23 23:24:31 CDT 2016 601872 91356 Thu Jun 23 23:24:36 CDT 2016 601872 91356 Thu Jun 23 23:24:41 CDT 2016 608116 101832 Thu Jun 23 23:24:46 CDT 2016 612212 104116 Thu Jun 23 23:24:51 CDT 2016 612284 105764 Thu Jun 23 23:24:56 CDT 2016 614332 106972 Thu Jun 23 23:25:02 CDT 2016 614332 106972 Thu Jun 23 23:25:07 CDT 2016 614332 106972 Thu Jun 23 23:25:12 CDT 2016 614332 107124 Thu Jun 23 23:25:17 CDT 2016 614332 107124 Thu Jun 23 23:25:22 CDT 2016 614332 107328 Thu Jun 23 23:25:27 CDT 2016 614332 107332 Thu Jun 23 23:25:32 CDT 2016 614332 107332 Thu Jun 23 23:25:37 CDT 2016 614332 107332 Thu Jun 23 23:25:42 CDT 2016 614332 107332 Thu Jun 23 23:25:47 CDT 2016 614332 107332 Thu Jun 23 23:25:52 CDT 2016 614332 107340 Thu Jun 23 23:25:57 CDT 2016 614332 107340 Thu Jun 23 23:26:02 CDT 2016 614332 107340 Thu Jun 23 23:26:07 CDT 2016 614332 107340 Thu Jun 23 23:26:12 CDT 2016 614332 107340 Thu Jun 23 23:26:17 CDT 2016 614332 107340 Thu Jun 23 23:26:22 CDT 2016 614332 107340 Thu Jun 23 23:26:27 CDT 2016 614448 107500 Thu Jun 23 23:26:32 CDT 2016 614448 107528 Thu Jun 23 23:26:37 CDT 2016 614448 107528 Thu Jun 23 23:26:42 CDT 2016 614448 107528 Thu Jun 23 23:26:47 CDT 2016 614448 107528 Thu Jun 23 23:26:52 CDT 2016 614448 107532 Thu Jun 23 23:26:57 CDT 2016 614448 107536 Thu Jun 23 23:27:02 CDT 2016 616496 108776 Thu Jun 23 23:27:07 CDT 2016 618572 111144 Thu Jun 23 23:27:12 CDT 2016 618580 112480 Thu Jun 23 23:27:17 CDT 2016 620648 113652 Thu Jun 23 23:27:22 CDT 2016 622888 115452 Thu Jun 23 23:27:27 CDT 2016 624796 116884 Thu Jun 23 23:27:32 CDT 2016 626896 120412 Thu Jun 23 23:27:37 CDT 2016 628964 121016 Thu Jun 23 23:27:42 CDT 2016 628992 121336 Thu Jun 23 23:27:47 CDT 2016 628992 121336 Thu Jun 23 23:27:52 CDT 2016 628992 121336 Thu Jun 23 23:27:57 CDT 2016 628992 121336 Thu Jun 23 23:28:03 CDT 2016 628992 121336 Thu Jun 23 23:28:08 CDT 2016 628992 121336 Thu Jun 23 23:28:14 CDT 2016 628992 121336 Thu Jun 23 23:28:19 CDT 2016 628992 121336 Thu Jun 23 23:28:24 CDT 2016 628992 121336 Thu Jun 23 23:28:29 CDT 2016 628992 121336 Thu Jun 23 23:28:34 CDT 2016 629028 121372 Thu Jun 23 23:28:39 CDT 2016 629028 121372 Thu Jun 23 23:28:44 CDT 2016 629028 121372 Thu Jun 23 23:28:49 CDT 2016 629028 121372 Thu Jun 23 23:28:54 CDT 2016 629028 121372 Thu Jun 23 23:28:59 CDT 2016 629208 121424 Thu Jun 23 23:29:04 CDT 2016 629160 123760 Thu Jun 23 23:29:09 CDT 2016 629160 123760 Thu Jun 23 23:29:14 CDT 2016 631208 126352 Thu Jun 23 23:29:19 CDT 2016 633996 127072 Thu Jun 23 23:29:24 CDT 2016 637532 130876 Thu Jun 23 23:29:29 CDT 2016 639664 133444 Thu Jun 23 23:29:34 CDT 2016 640356 135004 Thu Jun 23 23:29:39 CDT 2016 643884 137796 Thu Jun 23 23:29:44 CDT 2016 646016 139508 Thu Jun 23 23:29:49 CDT 2016 646016 139508 Thu Jun 23 23:29:54 CDT 2016 646764 140212 Thu Jun 23 23:29:59 CDT 2016 646196 139692 Thu Jun 23 23:30:04 CDT 2016 646196 139692 Thu Jun 23 23:30:09 CDT 2016 646196 139692 Thu Jun 23 23:30:14 CDT 2016 646196 139692 Thu Jun 23 23:30:20 CDT 2016 646196 139692 Thu Jun 23 23:30:25 CDT 2016 646196 139692 Thu Jun 23 23:30:30 CDT 2016 644148 139364 Thu Jun 23 23:30:35 CDT 2016 644148 139364 Thu Jun 23 23:30:40 CDT 2016 644148 139364 Thu Jun 23 23:30:45 CDT 2016 644148 139364 Thu Jun 23 23:30:50 CDT 2016 644148 139364 Thu Jun 23 23:30:55 CDT 2016 644148 139364 Thu Jun 23 23:31:00 CDT 2016 644148 139364 Thu Jun 23 23:31:05 CDT 2016 644148 139364 Thu Jun 23 23:31:10 CDT 2016 644148 139364 Thu Jun 23 23:31:15 CDT 2016 644148 139364 Thu Jun 23 23:31:20 CDT 2016 644148 139364 Thu Jun 23 23:31:25 CDT 2016 644148 139364 Thu Jun 23 23:31:30 CDT 2016 644148 139364 Thu Jun 23 23:31:35 CDT 2016 644148 139364 Thu Jun 23 23:31:40 CDT 2016 644148 139364 Thu Jun 23 23:31:45 CDT 2016 644148 139364 Thu Jun 23 23:31:50 CDT 2016 644148 139364 Thu Jun 23 23:31:55 CDT 2016 644148 139364 Thu Jun 23 23:32:00 CDT 2016 644148 139364 Thu Jun 23 23:32:05 CDT 2016 644148 139364 Thu Jun 23 23:32:10 CDT 2016 644148 139364 Thu Jun 23 23:32:15 CDT 2016 644148 139364 Thu Jun 23 23:32:20 CDT 2016 644148 139364 Thu Jun 23 23:32:25 CDT 2016 644148 139364 Thu Jun 23 23:32:30 CDT 2016 644148 139364 Thu Jun 23 23:32:35 CDT 2016 644148 139364 Thu Jun 23 23:32:40 CDT 2016 644148 139364 Thu Jun 23 23:32:45 CDT 2016 644148 139364 Thu Jun 23 23:32:51 CDT 2016 644148 139364 Thu Jun 23 23:32:56 CDT 2016 644148 139364 Thu Jun 23 23:33:01 CDT 2016 644148 139364 Thu Jun 23 23:33:06 CDT 2016 644148 139364 Thu Jun 23 23:33:11 CDT 2016 644148 139364 Thu Jun 23 23:33:16 CDT 2016 644148 139364 Thu Jun 23 23:33:21 CDT 2016 644148 139364 Thu Jun 23 23:33:26 CDT 2016 644148 139364 Thu Jun 23 23:33:31 CDT 2016 644148 139364 Thu Jun 23 23:33:36 CDT 2016 644148 139364 Thu Jun 23 23:33:41 CDT 2016 644148 139364 Thu Jun 23 23:33:46 CDT 2016 644148 139364 Thu Jun 23 23:33:51 CDT 2016 644148 139364 Thu Jun 23 23:33:56 CDT 2016 644148 139364 Thu Jun 23 23:34:01 CDT 2016 644148 139364 Thu Jun 23 23:34:06 CDT 2016 644148 139364 Thu Jun 23 23:34:11 CDT 2016 644148 139364 Thu Jun 23 23:34:16 CDT 2016 644148 139364 Thu Jun 23 23:34:21 CDT 2016 644148 139364 Thu Jun 23 23:34:26 CDT 2016 644148 139364 Thu Jun 23 23:34:31 CDT 2016 644148 139364 Thu Jun 23 23:34:36 CDT 2016 644148 139364