Reception Events not all appearing in logs

Hi Munin team,

After our performance testing runs we found the following issue with logging.

Reception events that are consumed by AER containers are not being written to the logs (gzipped files). It was noted that Reception containers were dying and then restarting constantly due to an error in file processing.

protocol-verifier-stack-aereception-2  | Program Error : Failed to create timer :  Resource temporarily unavailable
protocol-verifier-stack-aereception-2  |   Stack:
protocol-verifier-stack-aereception-2  |   #1   AEReception::FileControl.ProcessingFile:12

Steps to reproduce:

PV 1.0.0
Run 4 AEReception containers and at least 1 Verifier container
Send 500 events/s as separate files for 40 minutes (sent to reception incoming folder) – You would only likely need 10 mins of this as enough logs will have likely been produced by then
Job sequences are 33 events with 1 second interval between them
Set the size of Reception log files to be 5MB before they are compressed
Make sure the reception logs are not rotated so that all logs produced are kept and not deleted
Check the reception incoming folder to see if all events have been consumed
Gunzip all the compressed logs and run the following greps
- grep -E ‘reception_event_received’ Reception* | wc -l and confirm whether this number is the same as the total number of events sent
- grep -E ‘reception_event_written’ Reception* | wc -l and confirm whether this number is the same as the total number of events sent

xtuml / munin

Reception Events not all appearing in logs #143