Emails sent by the API are not received by free.fr email addresses

[asaunier]

For instance registration or email change emails. See the UI issue @desnoes has created: https://github.com/c2corg/v6_ui/issues/1763

It is maybe related to this function https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py#L53-L72 We used to send emails with transfer_encoding defaulting to quoted-printable but had to use base64 instead (using an explicit Attachment) because of encoding problems in messages received by hotmail addresses.

[asaunier]

It would be useful to figure out if emails sent by Discourse are also not delivered to free.fr addresses. If not, it probably means the problem is related to the third-party email sending service we use (SparkPost).

[asaunier]

According to https://forum.camptocamp.org/t/probleme-de-reception-des-mails-c2c-par-les-adresses-free-fr/198972 emails sent by Discourse (also using the SparkPost service) are correctly received by free.fr addresses. Which means the problem is not related to SparkPost but to the API email system, pointed above.

[stef74]

I have identified the error on sparkpost log : 550 mail dropped, bare LF found

After quick googling : we have found a similar

host free.fr[212.27.48.7}] said: 550 mail dropped, bare LF found

Sur internet, des internautes semblent avoir des problèmes de ce genre et disent que dans la classe: class.phpmailer.php il faut modifier une variable de "\n" à \r\n" Ce qui, si j'ai bien compris est un problème de terminaison de fin de ligne.

[asaunier]

mail dropped, bare LF found

Ok thanks for the hint!

I have read messages suggesting that free.fr rejects emails with an empty body (which is probably what "bare LF found" means?).

I suspect it is related to the way we build emails with pyramid_mailer: https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py#L57-L66 => we convert the email body to an attachment in order to set the transfer_encoding encoding to base64 instead of the default quoted-printable that caused encoding problem with hotmail recipients. Then all the message content is in the attachment, probably leaving the body itself empty.

I think we found no other way to set the transfer_encoding: https://docs.pylonsproject.org/projects/pyramid_mailer/en/latest/index.html#pyramid_mailer.message.Attachment

See the change at https://github.com/c2corg/v6_api/commit/0f46160d5b33188471156fd9c87c9391f05386ca#diff-4bfd085d779d3f100b5f48f467c82496

[julsbreakdown]

update : En changeant l'adresse SMTP, pour celui de gmail, les mails parviennent aux destinataires. L'erreur semble se situé au niveau du serveur SMTP psemail.epfl.ch

[asaunier]

L'erreur semble se situé au niveau du serveur SMTP psemail.epfl.ch

I don't think this SMTP is used. The demo and prod instances use the sparkpost.com service to send emails.

[julsbreakdown]

So sparkpost could be the culprit ?

[asaunier]

I don't think so. The Discourse (forum) notification emails are also sent using Sparkpost and are actually accepted by free.fr. Which is not the case of the emails sent by the v6 API. The problem seems to be around https://github.com/c2corg/v6_api/blob/master/c2corg_api/emails/email_service.py#L53-L72

[julsbreakdown]

So why would changing the mail host to gmail fix the problem?

[arnaud-morvan]

Regarding my tests, Bare LF error with Free was not related with Attachment done for hotmail.

I've replaced use of pyramid_mailer by using smtplib directly, and everything seems to work fine with free and hotmail. I've run send_emails_using_smtplib image in demov6, ready for testing before merge.

[arnaud-morvan]

Seems that Free SPAM policy has changed, I've just received a recover password email from demov6 in a free.fr mailbox.

[desnoes]

@arnaud-morvan : i have also tested and receive the email with my yahoo, gmail and free adresses :). Thanks. I have asked people from the association to do more tests with other email adresses. I would like to check it works with hotmail. We had some problem with this mail in the past.

[desnoes]

@arnaud-morvan @fjacon : did you review the code in order to prepare a PR ?

[fjacon]

@desnoes we were waiting for the association to do more tests. The code is already merged. Do you want us to deploy it on prod ?

[desnoes]

@fjacon: I already did the tests. I didn't know the code was merged. We will deply it on prod.