Memory leak in deqMany

[boriborm]

1. What versions are you using?

Platform: linux Node version: v14.17.6 Architecture: x64 node-oracledb version: 5.5.0 Oracle client version: 21.3.0.0.0

2. Is it an error or a hang or a crash?

Crash

3. What error(s) or behavior you are seeing?

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

4. Include a runnable Node.js script that shows the problem.

const oracledb = require('oracledb');
const conf = require('./config.json');
async function runQueue () {
  const pool = await oracledb.createPool(conf); // in conf connectString, user, password, events=true
  const conn = await pool.getConnection();
  const queue = await conn.getQueue('XXI.CLIENT_OUT');
  queue.deqOptions.visibility = oracledb.AQ_VISIBILITY_ON_COMMIT;
  queue.deqOptions.wait = 0;

  let c = 0;
  console.log(`start`);
  while (c<2000){
    await queue.deqMany(1000);  //queue hasn't messages, return undefined. More maxMessages value -> larger RSS leak
    const stats = process.memoryUsage ();
    console.log ( `count: ${c}, rss: ${stats.rss} `) ;
    c++;
  }
  console.log(`end`);
  await conn.close();
  await pool.close(1);
}
runQueue().catch(err=>console.log(err));

output:

start
count: 0, rss: 71262208 
count: 1, rss: 71585792 
count: 2, rss: 71585792 
count: 3, rss: 71852032 
...
...
count: 1989, rss: 203001856 
count: 1990, rss: 203001856 
count: 1991, rss: 203272192 
count: 1992, rss: 203542528 
count: 1993, rss: 203812864 
count: 1994, rss: 204083200 
count: 1995, rss: 204083200 
count: 1996, rss: 204083200 
count: 1997, rss: 204083200 
count: 1998, rss: 204083200 
count: 1999, rss: 204083200 
end

[cjbj]

Can you add code to create the queue and add runnable code do the enqueues, so we know what's in the queue?

[boriborm]

Queue is empty.

BEGIN 
  DBMS_AQADM.CREATE_QUEUE_TABLE(queue_table => 'MSG_QT',queue_payload_type => 'RAW'); 
  DBMS_AQADM.CREATE_QUEUE(queue_table => 'MSG_QT', queue_name => 'CLIENT_OUT);
  DBMS_AQADM.START_QUEUE('CLIENT_OUT); 
END

[cjbj]

@pvenkatraman another task for you :)

[boriborm]

Maybe bug in function dpiQueue_deqMany

When it run, then numMsgProps changed to 0 (no new messages, queue is empty)

I run code below and no memory leaks

static bool njsAqQueue_deqManyAsync(njsBaton *baton)
{
    njsAqQueue *queue = (njsAqQueue*) baton->callingInstance;

    baton->msgProps = calloc(baton->numMsgProps, sizeof(dpiMsgProps*));

    if (!baton->msgProps)
        return njsBaton_setError(baton, errInsufficientMemory);
/*
    if (dpiQueue_deqMany(queue->handle, &baton->numMsgProps,
            baton->msgProps) < 0)
        return njsBaton_setErrorDPI(baton);
*/
    baton->numMsgProps = 0; // Empty queue imitation
    return true;
}

Script in my first post output:

start
count: 0, rss: 60858368 
count: 1, rss: 61128704 
count: 2, rss: 61394944 
count: 3, rss: 61394944 
count: 4, rss: 61394944 
count: 5, rss: 61394944 
count: 6, rss: 61394944 
count: 7, rss: 61394944 
...
...
count: 1994, rss: 61669376 
count: 1995, rss: 61669376 
count: 1996, rss: 61669376 
count: 1997, rss: 61669376 
count: 1998, rss: 61669376 
count: 1999, rss: 61669376 
end

memory usage between 60858368 and 62414848

[pvenkatraman]

Thanks @boriborm will take a look.

[boriborm]

Bug in dpiQueue__deq when numMsgProps > 1 and payload is 'RAW', then run this function

status = dpiOci__aqDeqArray(queue->conn, queue->name,
                queue->deqOptions->handle, numProps, queue->buffer.handles,
                payloadTDO, queue->buffer.instances, queue->buffer.indicators,
                queue->buffer.msgIds, error);

This function run OCI function OCIAQDeqArray :(

When i change it to

   /*
        status = dpiOci__aqDeqArray(queue->conn, queue->name,
                queue->deqOptions->handle, numProps, queue->buffer.handles,
                payloadTDO, queue->buffer.instances, queue->buffer.indicators,
                queue->buffer.msgIds, error);
  */
  status = 0;
  *numProps = 0; // Empty queue imitation

no memory leak

[pvenkatraman]

Thanks, working with another team member on this issue. Will update soon.

[JoshuaCEnrico]

Is this problem solved in any specific version? Version less than 6 from here

[sharadraju]

@JoshuaCEnrico The memory profile with node-oracledb 6.x has changed slightly, because of the refactoring to use less C code in general, in addition to the newly introduced Thin mode. We will try to see if this can be reproduced in the 6.x version and get back to you.

[999max]

@sharadraju The issue still exist, even in the v6.5.1