Memory Leak after repeated inserts mixed with selects [CORE5319]

[firebird-automations]

Submitted by: Carlos Clicio (clicio)

Attachments: trace.txt tst_inc

I´m a long time firebird user with serveral production environments runnig Firebird 2.5.X . I´m extremally glad with Firebird.. When FB 3 was reeleased , i´ve got so exited with the great news available, and contiuned with the tests i´ve been doing since RC1. It all was fine, until i´ve got some tests, with long time runnig batch proccess in realy great databeses / tables , with thousands of insert/updates per proccess. Thats when my problem starts, after runnig some steps of the batch, the program abort abnormaly. I´ve started investigation and notices that the databese consumed ALL MEMORY AND SWAP , and run out of resorces , aboring as follows:

 Fri May 13 19:29:46 2016
    unable to allocate memory from operating system

After some time researching i have find a simple situation that reproduce the problem as follows.

I´ve noticed that my simple program that makes 2001 insertes, mixed with selects, makes that memory used by firebird process keeps growin , even after the program exits. So I put it in a linux for loop to reproduce a long batch with several interactions,with different programs images Even when i run a sweep in the process memory is not free.

Script to repeat the usage of the program : for i in $(seq 100); do ./tst_inc localhost sysdba masterkey /home/firebird/database.gdb ; done

Command to monitor memory usage after the script above: ps auxw | grep firebird | grep -v grep Result : firebird 4034 13.6 10.6 [[542176]] 431140 ? Sl 10:37 7:35 /opt/firebird/bin/firebird #⁠ final memory usage 540Mb after the loop

PS : the initial usage of the process is 90 Mb afer na isql in database.gdb;

PS2: I did the same with FB2.5 and there is NO PROBLEM

Program code:

// cat tst_inc.pas uses sysutils,IBConnection,sqldb; var FSqlConnection : TIBConnection; t : tsqltransaction; Q,Qry : TSqlQuery; id,i: integer;

function legenerator:integer; var Query : TSqlQuery; begin try Query := TSqlQuery.create(nil); Query.DataBase := FSqlConnection; Query.Sql.text := 'SELECT GEN_ID(GEN_LOG_EVENTOS,1) FROM RDB$DATABASE'; Query.Open; if not query.isEmpty then result := Query.Fields.Fields[0].AsInteger else raise exception.create(' Nao conesguiu alocar generator'); Query.Close; finally Query.Free; end; end;

begin FSqlConnection := TIBConnection.Create(nil); FSqlConnection.HostName := paramstr(1); FSqlConnection.UserName := paramstr(2); FSqlConnection.Password := paramstr(3); FSqlConnection.DatabaseName := paramstr(4); t := tsqltransaction.create(nil); FSqlConnection.transaction := t; FSqlConnection.Connected := True; t.Active := True; Q := TSqlQuery.create(nil); Q.DataBase := FSqlConnection; q.sql.add('select * from x); q.open; for i := 1 to 2001 do begin Qry := TSqlQuery.create(nil); Qry.DataBase := FSqlConnection; Qry.sql.add('INSERT INTO T (C1,c2,c3,c4,c5,c6,c7,c8,c9,c10) values'); if paramstr(5) = '' then Qry.sql.add('( ''cc'','+inttostr(legenerator)+',''...............................'',''dd'',''2016-01-01'',''.....................................'',''......................'',''.........................................'',''.................................'',''............................'' ) ') else Qry.sql.add('( ''cc'',0, ''...............................'',''dd'',''2016-01-01'',''.....................................'',''......................'',''.........................................'',''.................................'',''............................'' ) '); Qry.ExecSql; t.commit; Qry.Close; Qry.Free; end; q.close; q.free; t.active := false; FSqlConnection.close; end. // to compile fpc tst_inc

PS2: if i remov either one or the other select the firebird process doesn´t keep growing.

Table structure :

CREATE GENERATOR GEN_LOG_EVENTOS;

create table x ( a varchar(10 ) );

create table t ( C1 VARCHAR(1000) Not Null, C2 INTEGER Not Null, C3 BLOB, C4 VARCHAR(2000) Not Null, C5 DATE Not Null, C6 VARCHAR(1000) , C7 VARCHAR(1000) , C8 VARCHAR(1000) , C9 VARCHAR(1000) , C10 VARCHAR(1000) , C11 VARCHAR(10000) );

delete from t; select count (*) from t;

I´ll attach a fbmonitor output with 5 interactions

[firebird-automations]

Commented by: Carlos Clicio (clicio)

FBTrace with 5 interactions .

database = /home/firebird/database.gdb { enabled = true log_statement_start = true log_transactions = true

}

[firebird-automations]

Modified by: Carlos Clicio (clicio)

Attachment: trace\.txt \[ 13001 \]

[firebird-automations]

Modified by: Carlos Clicio (clicio)

summary: Memory Leak after repeated inserts mixed =\> Memory Leak after repeated inserts mixed with selects

[firebird-automations]

Commented by: Carlos Clicio (clicio)

The binnary file if need to run again. Linux 32 bits ELF binnary firebird 2.5 library

[firebird-automations]

Modified by: Carlos Clicio (clicio)

Attachment: tst\_inc \[ 13002 \]

[firebird-automations]

Commented by: @hvlad

Could you try latest snapshot build of Firebird 3.0.1 ?

[firebird-automations]

Commented by: Carlos Clicio (clicio)

Dear sir, i´ve tested with Snapshot version. And i have a good and a bad new. The good one is that memory growth stoped in my test. So i´ve continued with my application test. Thats where the bad new come. Everthing was fine , running my long batch programs, until i started a program and it hangs in the midle, therefore firebird refused sql commands like selectin mon$statemntes in isql and refusd connections sometimes. I´ve investigatede a little bit more and one behavior that is a little bit differnent is that this last program inserts a huge amount of records and between each it gives two commit_retaining. So that Oldest Transaction and Newest difference was about 500000. In firebird 2.5, where my application runs today, it also grows this way but as soon as the program ends there is a automatic commit of this gap of transactions. Could you once again , please, help me ?