Suddenly high memory usage

[andreasbaumann]

The service works normally using some 600MB of residental memory, then memory usage suddenly goes through the roof. Happened on a 64-bit machine, memory usage goes up to 2.5 GB on a machine with 4 GB RAM. The process has to be killed to get the machine back.

[andreasbaumann]

 4063 xxx   20   0 3079.0m 2.455g   0.0 64.6   9:37.68 T ./strusWebService -c config.js             

Then the machine hung.. :-)

[andreasbaumann]

Happend on Archlinux, the same process runs on Centos 6 stable at 800 MB residental memory. I'm really puzzled..

[andreasbaumann]

One idea seem to be CPPCMS settings:

service.worker_threads

The number of worker threads per process. Default is 5 * number of CPUs. For example quad core it would be 20 threads.

service.worker_processes

The number of forked worker processes. This option is relevant only for POSIX platforms.

Values:

0 means no fork is executed, default
1 means that one child process is forked and the parent supervises and and would restart if in case of crash.
>1 several processes are forked and try to accept incoming connections.

Let's try a 1 process, 1 thread version and see if we have a "duplicating memory on duplicate access" problem.

[andreasbaumann]

Another issue is, where do we store handles to strus objects and are they all in the global context? What happens if we have more than one strusWebService object, which seems to be the default policy of appilcation in cppcms:

service.applications_pool_size

User application objects are generally cached in special pool for future faster reuse, this parameter defines maximal number of applications that can be cached there.

By default it is twice of size of service.worker_threads.

[andreasbaumann]

 {"execution_time":0.000122504,"result":"ok","stats":{"nof_docs":1210}}

Good. And memory is at:

25651 xxx   20   0 1666.1m 1.272g  0.0 33.5   0:03.04 S ./strusWebService -c config.js              

[andreasbaumann]

What's the default for worker_processes? Clearly anything bigger than one results in memory usage per process!

[andreasbaumann]

worker_processes = v.get("service.worker_processes",0);

default seems to be 0.

[andreasbaumann]

tried with 0/24, 1/24, 1/1. I cannot see a difference in behaviour when setting the configuration options. What happens if they are not set?

[andreasbaumann]

Setting worker_processes > 1 leads to:

2016-10-31 13:33:13; strusWebService, error: error creating storage client: error creating storage client: failed to create database client: error creating database client: failed to open key value store database: IO error: lock ./storage/A/LOCK: Resource temporarily unavailable (master.cpp:97)

If we support worker processes, then one process must handle one or many indexes, but every index must be managed by exactly one process. This would be nice for user isolation in different indexes.

[andreasbaumann]

setting applications_pool_size to > 1 leads to more than one strusWebService object. This object has no state (only constant member or references/pointers to other objects). As long as the final methods for closing transactions are properly synchronized, nothing bad should happend.

[andreasbaumann]

 3382 baumann   20   0 3606.1m 2.473g  0.0 65.1   0:04.72 t strusWebService                             
 3383 baumann   20   0 3606.1m 2.473g  0.0 65.1  23:37.01 t strusWebService                             
 3974 baumann   20   0 3606.1m 2.473g  0.0 65.1   1:27.10 t strusWebService          

gdb shows:

* 1    Thread 0x7fb38e5d2740 (LWP 3382) "strusWebService" 0x00007fb38b36add3 in epoll_wait ()
   from /usr/lib/libc.so.6
  2    Thread 0x7fb384144700 (LWP 3383) "strusWebService" 0x00007fb38ae6a10f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
  3    Thread 0x7fb382942700 (LWP 3974) "strusWebService" 0x00007fb38ae6a10f in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0

which is not very logical, so threads are polling on I/O or sleeping and the thing just eats memory?!

config.js is:

        "worker_processes" : 0,
        "worker_threads" : 1,
        "applications_pool_size" : 1