AtlasOfLivingAustralia / alerts

Alerts services
https://alerts.ala.org.au
Other
1 stars 6 forks source link

Email alerts failed to be sent out at 9:39 on Wednesday #211

Open qifeng-bai opened 2 months ago

qifeng-bai commented 2 months ago

Email alerts are likely to be successfully sent out on other days.

May 1 09:30:58 aws-alerts-test-2022 postfix/smtp[368269]: 67DDC7EBFA: to=<alerts@ala.org.au>, relay=smtp-relay.gmail.com[142.251.175.28]:587, delay=2.8, delays=0/0/2.8/0, dsn=4.7.0, status=deferred (SASL authentication failed; server smtp-relay.gmail.com[142.251.175.28] said: 454-4.7.0 Too many login attempts, please try again later. For more information,?454-4.7.0 go to?454 4.7.0 https://support.google.com/mail/answer/7126229 dc21-20020a05690c0f1500b0061b1e8153easm1475111ywb.34 - gsmtp)

qifeng-bai commented 1 month ago

Before the scheduled task run at 9:30, there are a number of emails which were sent from alerts@ala.org.au to alerts@ala.org.au, which caused the 'too many login attempts' error.

in mail.log.1: The first error of 'limit exceeded' happened at 06:49:41 on Apr 30, an email from alerts@ala.org.au to systems@ala.org.au

Since 06:49 - ( the first error stored in logs) , emails was continuously sent to alerts@ala.org.au every 3-5 seconds. 'said: 550-5.4.5 Daily SMTP relay limit exceeded for user'

and then, it started causing "Too many login attempts" after 07:04:19

in mail.log.2, the first error happened at Apr 23 06:05

Apr 23 06:05:29 aws-alerts-test-2022 postfix/smtp[130957]: 8AD767EBD5: to=<systems@ala.org.au>, relay=smtp-relay.gmail.com[74.125.68.28]:587, delay=3.9, delays=0.02/0.04/3.3/0.49, dsn=5.4.5, status=bounced (host smtp-relay.gmail.com[74.125.68.28] said: 550-5.4.5 Daily SMTP relay limit exceeded for user. For more information on SMTP 550-5.4.5 relay sending limits please contact your administrator or go to 550 5.4.5

After reviewing 3 logs, there is a common thing:

The error of 'relay limit exceeded' happened after an email was sent to systems@ala.org.au and alerts@ala.org.au The error happened around 6:00 am

qifeng-bai commented 1 month ago

Further analysis:

When we send alerts out, the client is 'localhost' :

Apr 22 13:48:02 aws-alerts-test-2022 postfix/smtpd[111345]: connect from **localhost**[127.0.0.1]
Apr 22 13:48:02 aws-alerts-test-2022 postfix/smtpd[111345]: 510C27EB76: client=localhost[127.0.0.1]

However, the emails to alerts@ala.org.au and systems@ala.org.au, there is a cleanup process with ec2-13-54-34-80.ap-southeast-2.compute.amazonaws.com , - same machine with aws-alerts-test-2022 , ? why is not from localhost this time

Apr 23 06:05:25 aws-alerts-test-2022 postfix/cleanup[130955]: 8AD767EBD5: message-id=<20240422200525.8AD767EBD5@ec2-13-54-34-80.ap-southeast-2.compute.amazonaws.com>

qifeng-bai commented 1 month ago

Waiting for AWS SES plugin @brucehyslop