Create message pre-processor for split/join

[tofu-rocketry]

VAC creates single record messages that SSM then duly sends.

Add option to turn on agglomerator, which should be a separate script as it handle some parsing of the messages.

It should check the message header and agglomerate any messages of the same type until it reaches the per message limit (1000) or a new type of message, then send it on or write it back out depending on whether this is just a pre-processing step or part of the sending.

Possible issues

• What to put in the empaid label if we're agglomerating multiple messages?
• Failures in the middle of agglomerating.
• What to write out if saving sent messages? (Guess that should be the agglomerated message.)

See also #50.

[tofu-rocketry]

This may well need to be extended to a general pre-processor for messages - AMS seems to have limits on message size that cloud sites hit when they try to do an initial republish, so we should perhaps check message sizes and split them up if too large. This means that SSM will need some basic awareness of APEL message headers and breaks at least so that it can split/join them correctly.

[tofu-rocketry]

Message headers are listed here: https://github.com/apel/apel/blob/dev/apel/db/__init__.py

Basic regex for all the headers would be ^APEL(?:-[a-z]+)+-message: v[0-9].[0-9]$

[tofu-rocketry]

Only combine messages if the header is exactly the same, up to a limit of, say, 500 records in one message.

Example of combining two individual messages:

APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%

APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%

To:

APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%