Multiple containers failing in longer runs

[FreddieMatherSmartDCSIT]

Protocol Verifier containers are frequently failing and restarting. In longer runs with multiple AEReception containers and multiple AEOSVDC containers this seems to lead to AEReception failing completely and then large amounts of file processing errors in AEOrdering.

We have plotted metrics as the test progresses (grafana graphs show BST):

• CPU usage over time
• Memory usage over time
• File processing failures
• Number of containers running

CPU usage

Memory usage

AEOrdering file processing errors

Containers running

Steps to reproduce:

• PV1.0.0
• Run 4 AEReception containers and 2 Verifier Containers
• Send 500 events/s for 40 mins (sent to reception incoming folder)
• Job sequences are 33 events with 1 second interval between them
• Track CPU usage of containers
• Track Memory usage of containers
• Check total number of aeordering_file_processing_failure (plot as histogram with x-axis timestamp)
• Check docker logs for container failures and restarts

[cortlandstarrett]

fixed in main as of 30 oct 2023