Memory Consumption Problem in 2.1.0

after upgrading today to 2.1.0 (on ubuntu 16.04) We noticed a permanently increasing memory consumption in the run. This resulted in many cases in out of memory errors. We had to switch back to version 2.0.3 (We tested 2.0.1, this version was fine too)

We are streaming and decoding from a testfile with 600MB and 2Mio Messages. Version 2.0.1 and 2.0.3 needed 4-6 MB php-memory for streaming through the whole file. Version 2.1.0 increases memory with every message until its fails at 750MB memory limit!

php -i of failure version: msgpack extension Version => 2.1.0 header Version => 3.2.0

We used execute, unpack, reset ... see code below:

protected function getMessages__php_msgpack($messageCount = 1){
    $messages = array();
    if( $this->filePointer ){
        $continue = true;
        while( $continue ){
            $offset = 0;
            $this->unpacker->reset();
            $executeResult = $this->unpacker->execute($this->buffer, $offset);
            // since msgp-version 0.5.8 execute is true even if the msg-pack string is incomplete, but then offset is greater then buffer-size
            if( $executeResult AND $offset<=strlen($this->buffer) ){ 
                $this->buffer = substr($this->buffer, $offset);
                $messages[] = $this->unpacker->data();
                if( count($messages)==$messageCount ){
                    $continue = false;
                }
            } else{
                // buffer not decoded -> feed more
                if( !feof($this->filePointer) ){
                    // feed the buffer
                    $this->buffer .= fread($this->filePointer, $this->readBufferSize);
                } else{
                    // end of file
                    $continue = false;
                    $this->endOfFile = true;
                    if( $this->buffer!='' ){
                        trigger_error(__METHOD__.' buffer not empty at eof:'.Waps_Helper_String::getStringRepresentationOfBinary($this->getBuffer()), E_USER_WARNING);
                        $this->errorOccurred = true;
                    }
                }
            }
        }
    }
    return $messages;
}

msgpack / msgpack-php

Memory Consumption Problem in 2.1.0 #147