[FEATURE] Sample docs in notification message for deduped, and completedAlerts

Is your feature request related to a problem? PR https://github.com/opensearch-project/alerting/pull/1450 added support for printing sample documents for alerts that were generated during the execution of a bucket level monitor. However, this implementation only supports printing sample docs for newAlerts; the execution context for bucket level monitors can also include dedupedAlerts, and completedAlerts.

What solution would you like? There is a request to enhance this feature to support returning sample docs for dedupedAlerts, and completedAlerts as well. https://forum.opensearch.org/t/alerting-sample-documents-for-ctx-dedupedalerts-and-ctx-completedalerts/18695/6

Do you have any additional context? A possible starting point for exploration could involve leveraging the relatedDocIds attribute that's already present in the alert data model (link) to store the docIDs, and concrete index names for sample docs (e.g., storing a list of strings like index-name|doc-id) in the alert document itself. Then perform an mget request to retrieve those sample docs just before sending notification messages. Capturing these sample doc references at the time of alert generation could provide a "snapshot" view of what the index data looked like during that monitor execution; however, we'll also want to consider how best to handle situations when some/all of the sample docs are no longer available to retrieve via mget (e.g., the concrete index containing that doc was deleted).

I believe the relatedDocIds is currently only used by document level monitors, but further investigation will be needed to confirm that there would be no adverse interactions with repurposing the attribute in this way. We'll also want to confirm that this is the most efficient means of printing sample docs for dedupedAlerts, and completedAlerts.

opensearch-project / alerting

[FEATURE] Sample docs in notification message for deduped, and completedAlerts #1519