Using multiprocess pool to address the OOM issue

splunk / eventgen

Splunk Event Generator: Eventgen

Apache License 2.0

376 stars 180 forks source link

Closed li-wu closed 4 years ago

li-wu commented 5 years ago

Previously, when using multiprocess mode with 20 generatorWorkers, the process will be killed as Out of memory error.

After adopting pool with maxtasksperchild, the processes are now down for OOM after more than 24 hours.

Data ingestion with perdayvolume=600:

Process status and memory usage:

li-wu commented 5 years ago

I will check why the modular input test case failed.

li-wu commented 5 years ago

Running more than three days with stable output and without process down. And average memory usage is about 220MB per process.

li-wu commented 5 years ago

I will check why the modular input test case failed.

All tests are passed now.