Probably it ocurred memory leak after the select statement using query or fetch apis

[DoctorSheng]

• Operating System Name: Linux version 3.10.0-1160.el7.x86_64 Red Hat 4.8.5-39

• db2level output from clidriver if in use:

• Target Db2 Server Version:

• node.js Version: v8.10.0

• ibm_db version: v2.6.0、v2.5.1、v3.0.0

• For non-Windows, output of below commands: uname Linux uname -m x86_64

• Value of below environment variables if set: IBM_DB_HOME: /home/ap/zhwd/clidriver PATH:/home/ap/zhwd/clidriver/bin:/home/ap/zhwd/clidriver/adm:/home/ap/zhwd/clidriver/lib:/home/ap/zhwd/clidriver/bin:/home/ap/zhwd/clidriver/adm:/home/ap/zhwd/clidriver/lib:/home/ap/zhwd/clidriver/bin:/home/ap/zhwd/clidriver/adm:/home/ap/zhwd/clidriver/lib:/home/ap/zhwd/ZJUAPServer/script/ibm_db-2.5.1/node_modules/.bin:/home/ap/zhwd/clidriver/bin:/home/ap/zhwd/clidriver/adm:/home/ap/zhwd/clidriver/lib:/usr/lib64/qt-3.3/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/openssh/bin:/usr/local/sbin:/home/ap/zhwd/node-v8.10.0-linux-x64/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/lib/node_modules/pm2/bin:/home/ap/zhwd/.local/bin:/home/ap/zhwd/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/lib/node_modules/pm2/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/lib/node_modules/pm2/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/bin:/home/ap/zhwd/node-v8.10.0-linux-x64/lib/node_modules/pm2/bin LIB/LD_LIBRARY_PATH/DYLD_LIBRARY_PATH:/home/ap/zhwd/clidriver/lib::/home/ap/zhwd/nginx/CertServer/lib:/home/ap/zhwd/instantclient:/home/ap/zhwd/nginx/openresty/luajit/lib:/home/ap/zhwd/ZJUAPServer/appserver/modules/extra

• Test script to reproduce the problem.

• For installation related issue, complete output of npm install ibm_db command.

• For connection related issue, update database connection info in ibm_db/test/config.testConnectionStrings.json file and share complete output of below commands:

  • cd .../ibm_db
  • npm install
  • node test/test-basic-test.js

Steps to Reproduce:

1. ibmdb.open
2. connection.query ( return 5120 rows )
3. Then something occurred like memory leak , even though i called global.gc() or
4. ibmdb.open
5. stmt = connection.prepareSync
6. result = stmt.executeSync()
7. data = result.fetchAllSync()
8. result.closeSync()
9. Then something occurred like memory leak , even though i've called global.gc() and result.closeSync(). The memory(rss) rised rapidly only after fetchAllSync().

[bimalkjha]

@DoctorSheng Please update database connection info in ibm_db/test/config.testConnectionStrings.json file and save it. Then cd to ibm_db/test directory and run node test-fetch-apis.js. Do you see any issue with this test file? We have executed and do not see any memory leak issue. It is using all the APIs as you mentioned in above post. Please share your test program that reproduced the issue or modify ibm_db/test/test-memory-leaks-new-objects.js file to reproduce the issue and share here. We have executed node --expose-gc test-memory-leaks-new-objects.js command and it works fine. Thanks.

[DoctorSheng]

@bimalkjha Thanks for your reply !! I've done two test which you suggest above or not. First is cd ibm_db/test/ and run node test-query-select.js, it returns 5161 rows : code below: var common = require("./common") , odbc = require("../") , db = new odbc.Database() , assert = require("assert") ; db.openSync(common.connectionString); db.query("select SERVERID,PROJECTID,PROJECTVERSION from SERVER_DOWNLOAD where 1=1", function (err, data) { console.log('2------------:', process.memoryUsage()) db.closeSync(); console.log('3------------:', process.memoryUsage()) console.log(data.length) assert.equal(err, null); //assert.deepEqual(data, [{ COLINT: '1', COLTEXT: 'some test' }]); }); console.log('1------------:', process.memoryUsage()) setInterval(function(){ console.log('no gc :', process.memoryUsage()) }, 30000)

result below [zjuap@SAzhwdapp02 test]$ node ./test-query-select.js 1------------: { rss: 36110336, heapTotal: 10829824, heapUsed: 5487608, external: 72620 } 2------------: { rss: 39079936, heapTotal: 11354112, heapUsed: 6757872, external: 8608 } 3------------: { rss: 39350272, heapTotal: 11354112, heapUsed: 6766552, external: 8608 } 5161 no gc : { rss: 39350272, heapTotal: 11354112, heapUsed: 6779592, external: 8608 } no gc : { rss: 39350272, heapTotal: 11354112, heapUsed: 6784592, external: 8608 }

[DoctorSheng]

Secondly, i'd run node test-fetch-apis.js , it returns 5161 rows after each select statement. I tracked the using of memory (rss) from begin to the end , and push it into an ARRAY . result bleow [ { rss: 30191616, heapTotal: 10829824, heapUsed: 5493088, external: 80812 }, { rss: 36499456, heapTotal: 10829824, heapUsed: 5519400, external: 80812 }, { rss: 37040128, heapTotal: 10829824, heapUsed: 5525088, external: 80812 }, { rss: 47468544, heapTotal: 16072704, heapUsed: 8861176, external: 16800 }, { rss: 47468544, heapTotal: 16072704, heapUsed: 8861360, external: 16800 }, { rss: 47468544, heapTotal: 16072704, heapUsed: 8862792, external: 16800 }, { rss: 48820224, heapTotal: 16072704, heapUsed: 7806584, external: 16800 }, { rss: 49090560, heapTotal: 16072704, heapUsed: 8198720, external: 16800 }, { rss: 49090560, heapTotal: 16072704, heapUsed: 8199160, external: 16800 }, { rss: 49172480, heapTotal: 16596992, heapUsed: 8011144, external: 16800 }, { rss: 49172480, heapTotal: 16596992, heapUsed: 8011328, external: 16800 }, { rss: 49438720, heapTotal: 16596992, heapUsed: 8026520, external: 16800 }, { rss: 49709056, heapTotal: 16596992, heapUsed: 8046792, external: 16800 } ] after 30 seconds { rss: 49709056, heapTotal: 16596992, heapUsed: 8082112, external: 16800 } after 60 seconds { rss: 49709056, heapTotal: 16596992, heapUsed: 8086560, external: 16800 }

[DoctorSheng]

nodejs : v8.10.0 node-ibm_db: v2.6.2 target-db2 : v10.7

[bimalkjha]

@DoctorSheng Please use ibm_db v3.1.0 and then do tests. I think we have fixed few memleak issue post v2.6.2. Better to test using the latest version. Thanks.

[DoctorSheng]

Compilation environment： node : v8.10.0 python: 2.7 gcc: 9.3.0 clidriver: v11.5 node-ibm_db:v3.1.0 Using the newest version which named v3.1.0 to test (run node test-fetch-apis.js) , the result is:

 --------------------------------------------
Selected data using Promisified fetch() API: 
--------------------------------------------
Row1 =  { SERVERID: 'GS4TS143:3eb5fcb79ad6980564c35008925d131c',
  PROJECTID: 'RegularAccountCancel',
  PROJECTVERSION: '1.0.1' }
Row2 =  [ 'GS4TS143:3eb5fcb79ad6980564c35008925d131c',
  'ElectronicBankSign',
  '1.0.2' ]
Retrieve column data using getData API:
First Row Data = 
GS4T
S143:
Regul
1.0.1
Second Row Data = 
GS4T
S143:
El
1.0
------------------

---------------------------------------
Retrieve column data using async-await:
---------------------------------------
First Row Data = GS4TS143:, Regul,  1.0.1
Second Row Data = GS4TS143:, Elect,  1.0.2
------ return from testAsync --------

Connection Closed.
---------------------------------------------------

Memeory has been tracked like this

[ { rss: 30588928,
    heapTotal: 10829824,
    heapUsed: 5953192,
    external: 84863 },
 { rss: 36843520,
    heapTotal: 10829824,
    heapUsed: 5996520,
    external: 84863 },
  { rss: 37380096,
    heapTotal: 10829824,
    heapUsed: 6003000,
    external: 84863 },
  { rss: 48533504,
    heapTotal: 16072704,
    heapUsed: 7452072,
    external: 8608 },
  { rss: 48803840,
    heapTotal: 16072704,
    heapUsed: 7452256,
    external: 8608 },
  { rss: 48803840,
    heapTotal: 16072704,
    heapUsed: 7467600,
    external: 8608 },
  { rss: 48803840,
    heapTotal: 16072704,
    heapUsed: 9250376,
    external: 8608 },
  { rss: 48803840,
    heapTotal: 16072704,
    heapUsed: 9642792,
    external: 8608 },
  { rss: 48803840,
    heapTotal: 16072704,
    heapUsed: 9643232,
    external: 8608 },
  { rss: 49180672,
    heapTotal: 17121280,
    heapUsed: 8645552,
    external: 8608 },
  { rss: 49180672,
    heapTotal: 17121280,
    heapUsed: 8645736,
    external: 8608 },
  { rss: 49438720,
    heapTotal: 17121280,
    heapUsed: 8670344,
    external: 8608 },
  { rss: 49438720,
    heapTotal: 17121280,
    heapUsed: 8759488,
    external: 8608 },
  { rss: 49438720,
    heapTotal: 17121280,
    heapUsed: 8759928,
    external: 8608 },
  { rss: 49704960,
    heapTotal: 17121280,
    heapUsed: 8936144,
    external: 8608 } ]
// after 30 s 
no gc : { rss: 49704960,
  heapTotal: 17121280,
  heapUsed: 8974864,
  external: 8608 }
// after 60 s
no gc : { rss: 49704960,
  heapTotal: 17121280,
  heapUsed: 8979472,
  external: 8608 }

[DoctorSheng]

running node ./test-query-select.js result

[zjuap@SAzhwdapp02 test]$ node ./test-query-select.js 
1------------: { rss: 30547968,
  heapTotal: 10829824,
  heapUsed: 5880776,
  external: 84863 }
Result.length from testColAlias =  5161
2------------: { rss: 39501824,
  heapTotal: 11354112,
  heapUsed: 7051912,
  external: 8608 }
Result.length from testDupColNames=  5161
3------------: { rss: 41390080,
  heapTotal: 18169856,
  heapUsed: 7924088,
  external: 8608 }
3------------: { rss: 41660416,
  heapTotal: 18169856,
  heapUsed: 7938832,
  external: 8608 }
All tests executed.
//after 30s
no gc : { rss: 41660416,
  heapTotal: 18169856,
  heapUsed: 7949720,
  external: 8608 }
//after 60s
no gc : { rss: 41660416,
  heapTotal: 18169856,
  heapUsed: 7954672,
  external: 8608 }
// after 90s
no gc : { rss: 41660416,
  heapTotal: 18169856,
  heapUsed: 7958512,
  external: 8608 }

source code

let common = require("./common")
  , ibmdb = require("../")
  , assert = require("assert")
  ;

let conn = null;
let data = null;

main();

async function main() {
  conn = await ibmdb.open(common.connectionString);
  await testColAlias();
  console.log('2------------:', process.memoryUsage())
  await testDupColNames();
  console.log('3------------:', process.memoryUsage())
  await conn.close();
  console.log('3------------:', process.memoryUsage())
  console.log("All tests executed.");
}

async function testColAlias() {
  try {
      data = await conn.query("select SERVERID,PROJECTID,PROJECTVERSION from SERVER_DOWNLOAD where 1=1");
  } catch (error) {
      assert.equal(error, null);
  }
  console.log("Result.length from testColAlias = ",data.length);
  //assert.deepEqual(data, [{ COLINT: '1', COLTEXT: 'some test' }]);
  return;
}

async function testDupColNames() {
  let sql = "select SERVERID,PROJECTID,PROJECTVERSION from SERVER_DOWNLOAD where 1=1";
  try {
      data = await conn.query(sql);
  } catch(error) {
      assert.equal(error, null);
  }
  console.log("Result.length from testDupColNames= ", data.length);
}

console.log('1------------:', process.memoryUsage())

setInterval(function(){
    console.log('no gc :', process.memoryUsage())
}, 30000)

[DoctorSheng]

I am so sorry that I forgot to update the comment. Finally , I try to update to the latest version(3.1.0) into the application, then to observe the rising of memory . Amazing, it was showed steady . I think , I have made some mistakes on unit test before . Thanks for your help so much !! The problem has been resolved.

[bimalkjha]

@DoctorSheng Thanks for the update and saving our time. Glad to know that latest driver has fixed your issue. Thanks.