search

ibm-messaging / mq-mqi-nodejs

Calling IBM MQ from Node.js - a JavaScript MQI wrapper
Apache License 2.0
79 stars 41 forks source link

Buffer Memory Increases when Polling for messages #156

Open krishnatejap opened 1 year ago

krishnatejap commented 1 year ago

Hi @ibmmqmet,

We are using this module to fetch messages. The application is containerized. The application code is written as follows:

  1. make a connection to the IBMMQ Queue Manager using mq.Connx and Open the Queue using mq.Open. The connection is now Open.
  2. Using while loop, attempt to read a message using mq.GetSync. In case there is a message, decode and process it. Sleep for "x" seconds.
  3. Continue 2 until the connection is lost, thereby making an attempt to reconnect and read the message - read from 1.

The size of the messages in our MQ is 100KB. We initialized the buffer with length 102400. Instead of allocating a new buffer for every read operation, we are initializing one at the time of connect and using it every time we try to read a message.

Problem

There seems to be a memory leak in the function call mq.GetSync which is causing the application memory to increase exponentially - since we have already initialized and using the same buffer memory for every read, I think this is unexpected.

Can you please help me here?

ibmmqmet commented 1 year ago

I can't see any leak in the GetSync processing. I wrote a program that does the same as you describe, and the buffer that gets passed to the C MQGET function is identical on each iteration. How much do you think is leaked on each loop? The 100K of the data buffer or something else (eg the size of the MQGMO structure, if you are allocating a new GMO every time).

krishnatejap commented 1 year ago

Hi @ibmmqmet ,

I'm sorry for replying late on this. So, I was looking at the values reported under process.memoryUsage().arrayBuffers. Here are the statistics when I connect to the MQ using Mutual TLS connection and poll the system for every 5 seconds.

image

I'm doing the following:

  1. Initialized a buffer of length 102400 bytes. Using the same buffer for reading the content from MQ.
  2. On every iteration, I'm initializing MQMD and MQGMO. The gmo has the following options set [MQC.MQGMO_NO_SYNCPOINT, MQC.MQGMO_NO_WAIT, MQC.MQGMO_CONVERT, MQC.MQGMO_FAIL_IF_QUIESCING].

I tried to use the same instance of MQMD and MQGMO but I wasn't able to read the messages after the first run.

Please let me know if you are looking at some other details. Thank you in advance.

krishnatejap commented 1 year ago

Hi @ibmmqmet ,

Did you get a chance to look into this?

Thanks, Krishna

ibmmqmet commented 1 year ago

I suspect this might be due to a known issue in the NodeJS engine. See https://github.com/nodejs/node-addon-api/issues/1140

Doing "something" asynchronous in the application seems to be a solution as there is not a simple answer being provided by Node itself. I'll see if there's something suitable that can be done in GetSync, but that might break the promise of it being a synchronous function. So an app change might be needed.

krishnatejap commented 1 year ago

Thank you @ibmmqmet. Regarding the app change you mentioned, is it something that I should do at my end. If so, can you please guide me here?