Open tofu-rocketry opened 6 years ago
This may well need to be extended to a general pre-processor for messages - AMS seems to have limits on message size that cloud sites hit when they try to do an initial republish, so we should perhaps check message sizes and split them up if too large. This means that SSM will need some basic awareness of APEL message headers and breaks at least so that it can split/join them correctly.
Message headers are listed here: https://github.com/apel/apel/blob/dev/apel/db/__init__.py
Basic regex for all the headers would be ^APEL(?:-[a-z]+)+-message: v[0-9].[0-9]$
Only combine messages if the header is exactly the same, up to a limit of, say, 500 records in one message.
Example of combining two individual messages:
APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%
APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%
To:
APEL-individual-job-message: v0.3
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%
Site: UKI-SOUTHGRID-BHAM-HEP
etc.
%%
VAC creates single record messages that SSM then duly sends.
Add option to turn on agglomerator, which should be a separate script as it handle some parsing of the messages.
It should check the message header and agglomerate any messages of the same type until it reaches the per message limit (1000) or a new type of message, then send it on or write it back out depending on whether this is just a pre-processing step or part of the sending.
Possible issues
empaid
label if we're agglomerating multiple messages?See also #50.