[Enhancement] Reduce number of query executions needed for enhanced alert notification message context

[AWSHurneyt]

This current implementation executes the monitor query multiple times; the first time on line 124 to collect the data for trigger evaluation, and then subsequent searches are executed for each triggered trigger in order to collect sample documents.

Ideally, we want to collect the sample documents in the call to collectInputResults on line 124 so we can avoid multiple queries as that will improve performance.

_Originally posted by @AWSHurneyt in https://github.com/opensearch-project/alerting/pull/1450#discussion_r1526680170_

[AWSHurneyt]

Reopening as this has not been implemented yet. This would improve the performance of the feature.

[dblock]

Catch All Triage - 1 2 3 4 5