Open espala opened 5 months ago
which OS version do you use?
Hello @espala,
Thank you for opening. I have pinged the team as well.
@opensearch-project/admin please re-route to https://github.com/opensearch-project/alerting-dashboards-plugin
which OS version do you use?
Hi, I actually wrote above. I am using Redhat version 8.8, OpenSearch and OpenSearch Dashboards version 2.11.1-1
[root@server6 ~]# yum list installed |grep opensearch
opensearch.x86_64 2.11.1-1 @opensearch-2.x
opensearch-dashboards.x86_64 2.11.1-1 @opensearch-dashboards-2.x
[root@server6 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)
Hello @espala,
Thank you for opening. I have pinged the team as well.
@opensearch-project/admin please re-route to https://github.com/opensearch-project/alerting-dashboards-plugin
@kavilla thank you. We will be waiting for information from you.
Hi @espala, Would you be able to provide the JSON payloads for the monitor, and the notification channel (scrubbed of any sensitive info of course)?
To retrieve the monitor JSON, you can run the following command in devtools.
GET _plugins/_alerting/monitors/_search
{
"query": {
"match" : {
"monitor.name": "Orion Disaster Error Monitor"
}
}
}
And from the logs provided above, it looks like the ID for your Smtp Channel
is Qzjsho0B2TBnxA0NWL9r
. The following command can be used to retrieve the JSON for that channel.
GET _plugins/_notifications/configs/Qzjsho0B2TBnxA0NWL9r
If it looks like the channel returned by that command isn't the correct one, you can use the following command to get all of the notification channels configured on the cluster; I would just need the JSON for your Smtp Channel
.
GET _plugins/_notifications/configs
@AWSHurneyt Thank you very much for your response and interest. I'm sorry I've had a rough weekend and I'm responding late.
I queried all the paths you forwarded and wrote them all below. Can you check it?
By the way, I changed the sensitive data in the output. It may be different from the information I gave when opening the case before. If there is a difference, please ignore those parts (mail address, domain, etc.)
query 1
GET _plugins/_alerting/monitors/_search
{
"query": {
"match" : {
"monitor.name": "Orion Disaster Error Monitor"
}
}
}
results
{
"took": 19,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 2.1507282,
"hits": [
{
"_index": ".opendistro-alerting-config",
"_id": "hq0Xl40BzHB4GwYe75E5",
"_version": 16,
"_seq_no": 81,
"_primary_term": 2,
"_score": 2.1507282,
"_source": {
"type": "monitor",
"schema_version": 8,
"name": "Orion Disaster Error Monitor",
"monitor_type": "query_level_monitor",
"enabled": true,
"enabled_time": 1707914606685,
"schedule": {
"period": {
"interval": 5,
"unit": "MINUTES"
}
},
"inputs": [
{
"search": {
"indices": [
"orion-disaster-*"
],
"query": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"from": "{{period_end}}||-7h",
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"format": "epoch_millis",
"boost": 1
}
}
},
{
"match_phrase": {
"level": {
"query": "error",
"slop": 0,
"zero_terms_query": "NONE",
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
}
}
}
}
],
"triggers": [
{
"query_level_trigger": {
"id": "CzSzp40BPraQSw1iuOEJ",
"name": "Email Trigger",
"severity": "1",
"condition": {
"script": {
"source": "ctx.results[0].hits.total.value > 1",
"lang": "painless"
}
},
"actions": [
{
"id": "notification915423",
"name": "act",
"destination_id": "Qzjsho0B2TBnxA0NWL9r",
"message_template": {
"source": """Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
}
}
]
}
}
],
"last_update_time": 1708491595830,
"data_sources": {
"query_index": ".opensearch-alerting-queries",
"findings_index": ".opensearch-alerting-finding-history-write",
"findings_index_pattern": "<.opensearch-alerting-finding-history-{now/d}-1>",
"alerts_index": ".opendistro-alerting-alerts",
"alerts_history_index": ".opendistro-alerting-alert-history-write",
"alerts_history_index_pattern": "<.opendistro-alerting-alert-history-{now/d}-1>",
"query_index_mappings_by_type": {},
"findings_enabled": false
},
"owner": "alerting"
}
}
]
}
}
Query 2 GET _plugins/_notifications/configs/
{
"start_index": 0,
"total_hits": 3,
"total_hit_relation": "eq",
"config_list": [
{
"config_id": "e3s6Wo0BJ92cGwouTc9p",
"last_updated_time_ms": 1706615786856,
"created_time_ms": 1706615786856,
"config": {
"name": "myaddress",
"description": "",
"config_type": "email_group",
"is_enabled": true,
"email_group": {
"recipient_list": [
{
"recipient": "myaddress@mydomain.com"
}
]
}
}
},
{
"config_id": "Qzjsho0B2TBnxA0NWL9r",
"last_updated_time_ms": 1708491388112,
"created_time_ms": 1707365652587,
"config": {
"name": "Test Email Notification Channel",
"description": "",
"config_type": "email",
"is_enabled": true,
"email": {
"email_account_id": "9Djrho0B2TBnxA0NZb6i",
"recipient_list": [],
"email_group_id_list": [
"e3s6Wo0BJ92cGwouTc9p"
]
}
}
},
{
"config_id": "9Djrho0B2TBnxA0NZb6i",
"last_updated_time_ms": 1708491434491,
"created_time_ms": 1707365590433,
"config": {
"name": "email-relay-server",
"description": "",
"config_type": "smtp_account",
"is_enabled": true,
"smtp_account": {
"host": "smtp.mydomain.com",
"port": 25,
"method": "none",
"from_address": "opensearch@mydomain.com"
}
}
}
]
}
Query 3 GET _plugins/_notifications/configs/Qzjsho0B2TBnxA0NWL9r
{
"start_index": 0,
"total_hits": 1,
"total_hit_relation": "eq",
"config_list": [
{
"config_id": "Qzjsho0B2TBnxA0NWL9r",
"last_updated_time_ms": 1708491388112,
"created_time_ms": 1707365652587,
"config": {
"name": "Test Email Notification Channel",
"description": "",
"config_type": "email",
"is_enabled": true,
"email": {
"email_account_id": "9Djrho0B2TBnxA0NZb6i",
"recipient_list": [],
"email_group_id_list": [
"e3s6Wo0BJ92cGwouTc9p"
]
}
}
}
]
}
Hello everyone,
I am using Opensearch cluster and my cluster is running smoothly. I don’t get any error messages or logs in the standard run cycle.
I wanted to define an alarm specifically, so I defined a new “Monitors” definition from the “Alerting” menu. Then I define “Trigger” and select my “Smtp Channel” definition.
Everything works correctly in the Alerting mechanism, it finds the word I filtered and runs the trigger function. But it cannot send an email using my smtp information I entered in the “Notification” section. Because it is trying to do smtp authentication.
But I selected the “Encryption Method” as “none” in the “Email Senders” section under the “Notifications” menu. When I click the “send test email” button both in the “Channel” I defined and in the Alerting > Monitors > Triggers I defined, I see that it sends emails. I can also see the emails in my inbox.
But when “Alerting > Monitors > Triggers > Action” automatically sends an email, it tries to establish smtp authentication connection.
I want to send email without smtp auth. There is also “none” option on OpenSearch Panel. I think we have such a possibility. But I see in the log that I get a 503 error. 503 here is an error code representing smtp auth.
I am using an internal smtp relay server and I am not using the smtp auth mechanism. I need to be able to send emails without smtp auth as the Opensearch system allows me.
I think this problem is a bug. How can I solve this problem?
My first OpenSearch forum case; https://forum.opensearch.org/t/about-mail-sending-problem-on-alerting-notification-section/17786
My second case detailing the problem; https://forum.opensearch.org/t/problem-sending-mail-without-smtp-auth-on-opensearch-critical/17993
screenshot of my smtp definition;
![Screen Shot 2024-02-21 at 08 17 14](https://github.com/opensearch-project/OpenSearch-Dashboards/assets/9979648/a2664863-1788-4817-9fa7-4d8f9a9d770a)
screenshot of my Alerting/Monitors definition;
![Screen Shot 2024-02-21 at 08 13 55](https://github.com/opensearch-project/OpenSearch-Dashboards/assets/9979648/a9de5c1b-ee8f-45b8-a97c-648246ccf7d0)
Log;
Error: Failed running action: OpenSearchStatusException[{"event_status_list": [{"config_id":"Qzjsho0B2TBnxA0NWL9r","config_type":"email","config_name":"Test Email Notification Channel","email_recipient_status":[{"recipient":"myaddress@mydomain.com","delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}],"delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}]}]
Opensearch and my linux versions
Related sections in my logs
2. Subject At the same time, when it can’t find the word I filtered for the alarm in “Alerting” > “Monitor”, it says that the “Trigger” and “Action” functions are working in the Alerts section and shows that it sent the email. But this is not true either.
My filter works correctly. I am looking for the word “error” in the log data for the last 3 hours. When it really can’t find the word “error” in the last 3 hours, the “History” section goes to green “no alerts”. This works correctly.
But when I change the status of the alarms under the “Alerts” heading at the bottom of the dashboard to “Completed”, it shows it as “Email Trigger”.
In this case, it does not seem to reflect an true result. Because even though there was an alarm trigger, the action gave an error and could not send the email.