Memory Usage

[numtel]

There is a major memory leak through all of the load test end-to-end cases but none of the unit test cases. I have been trying to narrow down the exact cause.

Following paragraph regarding command: npm run make && node test/load/ --case=common.getResultSetDiff

Using the currently published version with async/await, the getResultSetDiff() method does not seem to leak memory as seen in this 30 minute run over 24,440 iterations on random mutations of a result set with 500+ rows using this load test case.

Following paragraphs regarding command: node test/load/ --case=static

If query updates on notifications is disabled by commenting out this line, there is still a significant memory leak as shown in this run of 17 minutes.

Only if the _updateQuery() method is never called after initialization (instead of being called with no work to do as in the previous example) does the memory leak disappear. See this run of 17 minutes.

This is what led me to believe that the Promises and async/await were contributing to the memory leak. For a comparison between async techniques, see these memory usage graphs.

The trouble is that further runs on the end-to-end test cases show that switching to using Futures and callbacks does not help at all:

Test Case	Async Method	Results
static	Async/await (master branch)	8 mins run
spikes.manual	Async/await (master branch)	28 mins run
static	Futures and _resolveAll() (futures branch)	8 mins run
spikes.manual	Futures and _resolveAll() (futures branch)	28 mins run
spikes.manual	Futures and callbacks (updateQuery_callback branch)	26 mins run

Test case details:

• static (90 insert + 90 update per second)

  Command: node test/load/ --case=static

• spikes.manual (sinusoidal up to 45 insert + 45 update per second over ~10 second intervals)

  Command: node test/load/ --case=spikes.manual

Any contributions welcome to combat this problem.

[numtel]

There is an extreme disparity in memory usage between 380 rows per result set and 400 rows per result set when performing the query to grab refreshed a result set repeatedly over 15 minutes. (60889 iterations for 380 rows, 52727 iterations for 400 rows over 891 seconds.)

See new assignCount meta test case added in this commit:

node test/load/ --case=assignCount will run multiple cases with a range of assignment counts (allowing for different result set lengths) each for 15 minutes.

[numtel]

Running that same query from the previous comment (the same one used to refresh result sets in this package) but with 600+ rows per result set in a new (ultra simple) load test application shows that there is no memory leak in this 15 minute run. (54423 test queries and 46483 rows inserted on 120000 initial scores rows)

[numtel]

:open_mouth: :joy: :yum:

Memory leak finally under control! Just released, version 1.0.0 rewrites the entire package to use nothing but callbacks for asynchronous code. By ditching Promises and Babel throughout the library, the memory leak seems to have vanished!

See this result of a 22 minute run of the new load test running on 50 queries with 600+ rows.