Mail() function not working correctly in PHP 8.x

[hansherlighed]

Description

The following code:

<?php
mail("someone@example.com", "This is the subject", "This is the body");
?>

Resulted in this output of the DATA part of the communication from tcpdump on the receiving SMTP server when building container "FROM php:8.0.14-fpm-alpine" or "FROM php:fpm-alpine":

Received: by f3de6efeb3a0 (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:28:10 +0100\r\n
From: "Linux User" <www-data@f3de6efeb3a0>\r\n
Date: Sat, 12 Feb 2022 06:28:10 +0100\r\n
To: redacted@example.com\r
\r\n
Subject: This is the subject\r
\r\n
\r
\r\n
This is the Body\r
\r\n
\r\n

And this is the resulting mail headers of the mail on the receiving SMTP server:

Return-Path: <www-data@f3de6efeb3a0>
X-Envelope-To: redacted@example.com
Message-ID: <3966183578-2392@mail.example.com>
Received: from localhost ([127.0.0.1])
    by mail.example.com with SMTP
    for redacted@example.com;
    Sat, 12 Feb 2022 06:28:10 +0100
Received: by f3de6efeb3a0 (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:28:10 +0100
From: "Linux User" <www-data@f3de6efeb3a0>
Date: Sat, 12 Feb 2022 06:28:10 +0100
To: redacted@example.com

This is the output using the exact same environment, just that the container is build using "FROM php:7.4-fpm-alpine":

Received: by ae23725475da (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:27:33 +0100\r\n
From: "Linux User" <www-data@ae23725475da>\r\n
Date: Sat, 12 Feb 2022 06:27:33 +0100\r\n
To: redacted@example.com\r\n
Subject: This is the subject\r\n
\r\n
This is the Body\r\n

And this is the resulting mail headers of the mail on the receiving SMTP server:

Return-Path: <www-data@ae23725475da>
X-Envelope-To: redacted@example.com
Message-ID: <3966147578-2389@mail.example.com>
Received: from localhost ([127.0.0.1])
    by mail.example.com with SMTP
    for redacted@example.com;
    Sat, 12 Feb 2022 06:27:33 +0100
Received: by ae23725475da (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:27:33 +0100
From: "Linux User" <www-data@ae23725475da>
Date: Sat, 12 Feb 2022 06:27:33 +0100
To: redacted@example.com
Subject: This is the subject

The result is that when using PHP 8 there are some invalid "\r" which makes the mail format invalid and so the headers are not processed correctly on the receiving SMTP server. The result is that the email does not have a subject. I've also tried setting additional headers, but they are all invalid and not processed due to the invalid "\r". I've tried using both sendmail and ssmtp in the container. It yields the same result.

My php.ini setting are:

sendmail_path = '/usr/sbin/sendmail -t -i -S myserver:25'

Something has changed in the PHP mail() function between PHP 7 and PHP 8

PHP Version

PHP 8.0.14 and newer

Operating System

Docker php:8.0.14-fpm-alpine and php:fpm-alpine

[cmb69]

mail() now outputs proper email line endings (i.e. CRLF instead of LF). It looks like your MTAs convert that to CRCRLF, what would be a bug in those MTAs.

[hansherlighed]

mail() now outputs proper email line endings (i.e. CRLF instead of LF). It looks like your MTAs convert that to CRCRLF, what would be a bug in those MTAs.

The capture from tcpdump is taken on the PHP server, so the trace is captured when going FROM PHP to the MTA(it's the output from sendmail). So the CRCRLF seems to comes from PHP, not the MTA(unless you refer to sendmail as the MTA). If I read the specifications correctly, PHP should output CRLF directly after each line(which it does in PHP 7.4), but does not in PHP 8.

When you say MTA, do you mean sendmail? Do you mean that it is sendmail that is converting the input from PHP to CRCRLF? Is there anyway to see the raw output sent from PHP to sendmail?

[cmb69]

You can see the raw PHP output by e.g. writing to a file instead of using sendmail. I.e. set sendmail_path to something like tee mailBasic.out >/dev/null.

[hansherlighed]

Thanks @cmb69

I was able to get the raw output from the mail() function into a file using your advice. You are indeed right, the raw output from PHP 8 looks like this:

cat -e raw_mailoutput_php8.out

To: redatced@example.com^M$
Subject: This is the subject^M$
^M$
This is the Body^M$

and in PHP 7 it looks like this:

cat -e raw_mailoutput_php7.out

To: redacted@example.com$
Subject: This is the subject$
$
This is the Body$

This issue has actually been reported to PHP way back in 2002: https://bugs.php.net/bug.php?id=15841 The problem remains the same as reported in this bug. on UNIX systems, the line ending is expected to be "\n" as it is the native line ending for the system. Since PHP's mail() function on UNIX systems, is basically sending raw text output to a UNIX command(sendmail if you follow the documentation) the mail() function will not work as sendmail expects the line endings to be "\n" and NOT "\r\n". By making this new change to the mail() function so it always uses "\r\n" for the build in headers for Subject: and To:, you've actually made things worse, and made your documentation invalid. It states in the mail() documentation page:

Note:

If messages are not received, try using a LF (\n) only. Some Unix mail transfer agents (most notably [» qmail](http://cr.yp.to/qmail.html)) replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with [» RFC 2822](http://www.faqs.org/rfcs/rfc2822).

Following this note will cause the output from PHP8 to be a mix of "\r\n" and "\n" for the headers. The build in headers for "To:" and "Subject:" will still use "\r\n" even if you set the additional headers to use only "\n". running mail(redacted@example.com, "This is the subject", "This is the Body", "From: me@example.com\nContent-Type: text/plain"); results in:

cat -e raw_mailoutput_php8.out

To: redacted@example.com^M$
Subject: This is the subject^M$
From: me@example.com$
Content-Type: text/plain^M$
^M$
This is the Body^M$

The essence of the problem is that a UNIX command/program expects line endings to be "\n" and not "\r\n", and since the mail() function on UNIX systems, forwards/outputs to a UNIX command rather than talking to an SMTP server directly, it causes problems. Why can't the mail() function use the same line endings as PHP_EOL, so that on windows it becomes "\r\n" and unix "\n"? That would make the function more platform independent.

The solution when using the "php:fpm-alpine" image is to set "sendmail_path" to: '/usr/bin/dos2unix -u|/usr/sbin/sendmail -t -i'

[cmb69]

The essence of the problem is that a UNIX command/program expects line endings to be "\n" and not "\r\n", […]

That is irrelevant. The respective RFCs are Internet standards, and those mandate CRLF as line break, so MTAs should be able to deal with that regardless of the operating system they're running on. Consider to report this issue to the sendmail maintainers (if it still hasn't been fixed in the latest version).

[csandanov]

I stumbled upon the same issue recently.

The solution with dos2unix by @hansherlighed fixed it for me, thank you!

This issue happens both with busybox's and ssmtp's sendmail except with busybox's (unless you add verbosity) you will get this weird error sendmail: . failed when in fact it's the same RFC 2822 compliance error.

I don't know which SMTP server the OP used but I used OpenSMTPD, so possibly related https://github.com/OpenSMTPD/OpenSMTPD/issues/1135

[speller]

I just faced this issue in the PHP8.1 alpine image. It was fine in PHP7 images.

[speller]

Consider to report this issue to the sendmail maintainers

Why PHP contributors didn't do this knowing that alpine builds are failing one of the basic functions?

[cmb69]

Why PHP contributors didn't do this knowing that alpine builds are failing one of the basic functions?

Because it is not a PHP issue. mail() creates mails with CRLF line endings according to the standard; MTAs need to deal with this.

[pfletch101]

I have just been bitten by this on a site hosted by a commercial ISP, where I don't have root access, don't know precisely what distro is being used, and dos2unix is not installed, so I can't use the workaround. I'm afraid that I don't agree that this isn't a PHP issue. Previous PHP versions implemented mail() by communicating directly with the MTA. In doing so, they presumably correctly followed the RFCs requiring "\r\n" as the end of line characters. For your own reasons (and I suspect that they were quite good ones), you have now decided to use sendmail as an intermediary. sendmail is designed and intended to work at and from the Unix command line. It is therefore entirely reasonable for sendmail to expect header and other 'lines' to be terminated by the standard Unix "\n". If PHP chooses to use sendmail as its intermediary in sending mail, it behoves PHP to comply with its formatting rules.

[cmb69]

PHP uses a configurable MTA (e.g. sendmail) on non Windows platforms for ages; nothing has changed in this regard. What has changed, is that as of PHP 8.0.0, the mail() function produces standard conforming header line endings throughout (i.e. CRLF, instead of LF). The first standard RFC which specifies these line endings is RFC 822 from August 13, 1982. If there are any MTAs which still cannot deal with that standard, I'd blame them. And whether these MTAs run on Linux or any other platform is irrelevant; the use of CRLF is mandated for all platforms.

dos2unix is not installed, so I can't use the workaround

You don't even need dos2unix for a workaround; something like sed /\r\n/\n/g should work as well, if you're allowed to change the sendmail_path setting.

[speller]

@pfletch101 Nobody cares. PHP team has implemented a standard breaking the whole functionality and just saying "That's not our business to solve this". The Docker team who build the Docker image says "consider installing an alternative image" and closed the ticket. The sendmail bugtracker is dead and nobody responded at all: https://github.com/mirror/busybox/issues/58. In the end, we have a broken functionality, and everyone just ping-ponging users.

[speller]

@cmb69

PHP uses a configurable MTA (e.g. sendmail) on non Windows platforms for ages

Could you stop using sendmail since it doesn't follow the standard and:

I'd blame them

?

Does it look nonsense to blame an MTA for non-standard behavior and keep using it?

I think the PHP team should either fix the issue and revert back the compatibility with the tool that it "uses for ages" or stop using it and provide an alternative or make sure the team of the "tool used for ages" is aware of the changed protocol and release a version compatible with the updated PHP version. I can't see other options that keep the product quality high and do not break the basic functionality. Do you?

[cmb69]

Does it look nonsense to blame an MTA for non-standard behavior and keep using it?

PHP, despite the name of the INI setting, does not rely on sendmail (and never did). sendmail_path is just a command line to which the mail (as text) is piped, and which is then executed. You could even use some script (possibly written in PHP), which processes STDIN and "forwards" the mail to an SMTP server.

Note that this behavioral change is actually a bugfix (that bug affected other MTAs), namely that we sent mixed line endings (CRLF and LF), and that bug has finally been resolved, but we deliberately did that only for a major PHP version, because we were aware of the potential BC break.

Also note that it should be possible to allow the direct SMTP transfer, which is currently available only on Windows, on other platforms as well. A PR would be welcome.

[pfletch101]

The ability to send email is a fairly important functionality for a programming language whose primary purpose is to code website behaviors. It would seem that a choice to implement this with the use by means of an external intermediary (rather than using SMTP) should require that a reasonable effort be made to ensure that the implementation works and continues to work on most targeted systems. If you know that your chosen implementation has flaws and/or is not standards-compliant, the appropriate response would seem to be to change your implementation, not to expect the external intermediary to change its longstanding behavior, and certainly not to change your implementation in a way that will break functionality when the external intermediary predictably does not do so.

To say that the mail() function "does not rely on sendmail" may be technically true, for the reasons given, but I doubt whether more than 5% of the users of PHP know the details of how mail() works, nor should they need to! Only a very small minority would have the knowledge or ability to write (in essence) a replacement for sendmail.

[speller]

@cmb69

does not rely on sendmail (and never did)

This is not true. sendmail is used as the default MTA here and there and everywhere across many files in PHP sources.

So the situation is completely weird. The PHP team uses sendmail actively as the default MTA, intentionally breaks the compatibility with this MTA, and shifts the responsibility to users blaming MTAs that they don't follow standards. Moreover, the PHP team has broke the compatibility and didn't suggest any easy-to-use replacement.

The only issue is that users don't f**** care about standards, they need a working solution that brings results. And the PHP team broke this functionality. MTAs didn't change their behavior, they're not a part of this bug. So this is clearly the PHP responsibility to fix it back.

I think a proper solution will be to add a configuration key to switch between two types of line endings. As easy as that.

[michaelbrunnbauer]

Stumbled over this one today after upgrading to 8.x. The stubbornness of the PHP maintainers is astonishing. It should not have surprised me though, after all the pain they caused me and others by making a product with a life cycle of just 3 years.

[bukka]

I think this needs to be re-opened as we should not introduce a change that breaks compatibility with non conforming clients without an option to switch it back. I think it is a good thing to have conforming client by default but we should probably have some ini option to use non conforming behavior.

[bukka]

I changed it to Feature because technically this is part of UPGRADING and breaking change in the major version so we cannot consider it as a bug - it will also require introducing a new INI. If the patch looks self contained, we might potentially be able to get it to 8.2.

[bukka]

@cmb69 is it just this change? https://github.com/php/php-src/commit/6983ae7 . If so, it should be pretty easy to make it configurable...

[cmb69]

Yeah, it is just that change. But do we really need to make that configurable? sendmail has a -ba option, and it looks like this would enable proper handling of CRLF.

[bukka]

Ah ok, I didn't know that there is an option for that as the only suggestion before was just to use dos2unix which didn't sound to me like an acceptable work around and I haven't checked the sendmail docs. But if setting sendmail_path to /usr/sbin/sendmail -ba -t -i fixes the issue, then there's no point to introduce extra ini.

I will close it but if -ba doesn't work for anyone, we can re-open and look to the optionally disabling the change.

[bukka]

Actually if that's the case we should set it here as well: https://github.com/php/php-src/blob/f40c3fca88bd3a7ed516f05ba04999dcb3256ab2/main/main.c#L669

[cmb69]

Frankly, I haven't tried the option, since I'm usually working on Windows. Would be great if someone could actually try it out, and if it works, we should probably document it.

[bukka]

Yeah and also update it in https://github.com/php/php-src/blob/f40c3fca88bd3a7ed516f05ba04999dcb3256ab2/php.ini-production#L1098 and https://github.com/php/php-src/blob/f40c3fca88bd3a7ed516f05ba04999dcb3256ab2/php.ini-development#L1096

[bukka]

I will give it a try on the reported alpine image

[heiglandreas]

I just ran some tests:

$ echo '<?php mail("someone@example.com", "This is the subject", "This is the body");' > mailtest.php
$ docker run -v `pwd`:/var/www -w /var/www php:8 php -d sendmail_path="cat > ./test.8.log" mailtest.php
$ docker run -v `pwd`:/var/www -w /var/www php:7 php -d sendmail_path="cat > ./test.log" mailtest.php  
$ hexdump -c test.8.log
0000000   T   o   :       s   o   m   e   o   n   e   @   e   x   a   m
0000010   p   l   e   .   c   o   m  \r  \n   S   u   b   j   e   c   t
0000020   :       T   h   i   s       i   s       t   h   e       s   u
0000030   b   j   e   c   t  \r  \n  \r  \n   T   h   i   s       i   s
0000040       t   h   e       b   o   d   y  \r  \n                    
000004b
$ hexdump -c test.log
0000000   T   o   :       s   o   m   e   o   n   e   @   e   x   a   m
0000010   p   l   e   .   c   o   m  \n   S   u   b   j   e   c   t   :
0000020       T   h   i   s       i   s       t   h   e       s   u   b
0000030   j   e   c   t  \n  \n   T   h   i   s       i   s       t   h
0000040   e       b   o   d   y  \n                                    
0000047

The interesting part is that the PHP7 version uses \n to separate the Header lines, where PHP8 seems to use \r\n.

As no MTA whatsoever is involved in this test I assume we can ignore sendmail (or whatever people think is sendmail in their respective setup)

As the result from PHP8 is what is expected per RFC822 I assume we can say that the bug was fixed in general. Any issues with particular images need to be addressed in that image. Any issues with particular MTAs need to be addressed with that MTA. PHP is (after how many years now?) finally passing valid code towards whatever follows when calling mail

If anyone else can reproduce this or - even better - can not reproduce this, please post your findings here

[heiglandreas]

The results when using php:8-alpine instead of php:8 are the same. It is using the correct RFC822 conform line-ending.

[cmb69]

If anyone else can reproduce this or - even better - can not reproduce this, please post your findings here

There is even a test for this: https://github.com/php/php-src/blob/master/ext/standard/tests/mail/bug47983.phpt.

[michaelbrunnbauer]

Not all sendmail variants have the -ba option, e.g. qmail, netqmail, s/qmail.

[cmb69]

Not all sendmail variants have the -ba option, e.g. qmail, netqmail, s/qmail.

According to Wikipedia, qmail had it's final release 24 years ago; netqmail 15 years ago. I would presume that such mailers have other issues, which might render them irrelevant. s/qmail seems to be maintained, though. However, are these even affected by this CRLF issue? Or do they perhaps have another option to conform to the 40 year old specs?

[bukka]

s/qmail seems still maintained though so if it has got that issue and there's no -ba option, it's a good enough reason to introduce the switch.

I actually tested the sendmail v8.15.2 on Ubuntu 20.04 and sending email works fine there. It seems that just some variants might have got this issue.

[bukka]

The thing is that if those variants do not support -ba, we can't add it as a default.

[bukka]

I'm just reading the message sent above and it looks that the actual problem is with sSMTP which actually doesn't support -ba so it is not a solution by the look of it. See https://linux.die.net/man/8/ssmtp

[bukka]

So we probably need that INI that reverts it...

[michaelbrunnbauer]

Not all sendmail variants have the -ba option, e.g. qmail, netqmail, s/qmail.

According to Wikipedia, qmail had it's final release 24 years ago; netqmail 15 years ago. I would presume that such mailers have other issues, which might render them irrelevant. s/qmail seems to be maintained, though. However, are these even affected by this CRLF issue? Or do they perhaps have another option to conform to the 40 year old specs?

They are all affected and they are all alive. And there are many more variants of sendmail I am not familiar with. Stuff is not useless just because it is old. One does not always need newer features - especially on systems relaying mail to a smarthost. But please go ahead to the future and leave the past behind. I'll be smiling while the past catches up with the maintainers many many times in the next years.

[cmb69]

Stuff is not useless just because it is old.

Right. So why don't you just stick with PHP 7.4?

[michaelbrunnbauer]

Stuff is not useless just because it is old.

Right. So why don't you just stick with PHP 7.4?

You have no idea. I actually lost customers who would move somewhere else because they can keep using PHP 7.4 there. Most of the others had to be forced to upgrade their stuff - which was not easy for many of them. I know people who still use PHP 5.3. Screaming all day to them what a security risk that is doesn't change a thing. You have no idea. No idea at all.

[bukka]

Not all sendmail variants have the -ba option, e.g. qmail, netqmail, s/qmail.

According to Wikipedia, qmail had it's final release 24 years ago; netqmail 15 years ago. I would presume that such mailers have other issues, which might render them irrelevant. s/qmail seems to be maintained, though. However, are these even affected by this CRLF issue? Or do they perhaps have another option to conform to the 40 year old specs?

They are all affected and they are all alive. And there are many more variants of sendmail I am not familiar with. Stuff is not useless just because it is old. One does not always need newer features - especially on systems relaying mail to a smarthost. But please go ahead to the future and leave the past behind. I'll be smiling while the past catches up with the maintainers many many times in the next years.

Well we sometime need to stop supporting of some old versions and clients because it will become very difficult to make sure that the stuff still works there. In general we tend to do that when the support ends on the distributions that are still supported. For example currently the oldest distribution that we still support is RHEL 7 and its variants that are still alive (CentOS 7, Amazon Linux 2). That's how we decide to drop specific versions of the libraries.

This particular case is not such a case as it impacts many still supported emulations. However this problem is also causing some issue for other MTA as described in https://bugs.php.net/bug.php?id=47983 and PHP 8.0 was a good time to fix it. It was however omission to not keep the compatibility optionally which we should do if such change is done. However it's difficult to know that when making the change because it is very hard to actively test all MTA's. The only thing we can do is to add that option to the future version and learn from this that doing changes to mail output should require more testing and consider more MTA's.

[bukka]

I just created a PR https://github.com/php/php-src/pull/10191 that introduces an option to revert back to the previous behavior.

[michaelbrunnbauer]

BTW why are people pointing to rfc822? The introduction seems very clear to me:

Some message systems may store messages in formats that differ from the one specified in this standard. This specification is intended strictly as a definition of what message content format is to be passed BETWEEN hosts.

Note: This standard is NOT intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages.

[bukka]

BTW why are people pointing to rfc822? The introduction seems very clear to me:

Some message systems may store messages in formats that differ from the one specified in this standard. This specification is intended strictly as a definition of what message content format is to be passed BETWEEN hosts. Note: This standard is NOT intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages.

I see the point here as technically it is up to MTA to follow the RFC when sending the message. This is really just a program interface. On the other side, the current format had its issue so an attempt to fix them was probably in place. I would personally treat this as an omission but I'm not sure if requiring compliance was in place here. I have to say that after doing some research I haven't found anywhere any spec for sendmail interface that would mandate message format to follow the RFC 822, 2822 or 5322. Honestly I found very little info in sendmail docs but I might have missed something so if anyone else sees it somewhere, please share your findings.

[speller]

@bukka Thank you a lot for the fix!

[ElliotNB]

@bukka Thank you for the fix! I was also recently bitten by this issue. Do you know when your GH-8086 fix will make it into the next PHP 8.1.* release? I was inspecting the most recent commits contained in 8.1.15 and this fix doesn't appear to be included: https://github.com/php/php-src/compare/PHP-8.1.13...php-8.1.15

[bukka]

@ElliotNB The fix is not in 8.1 as it's somewhere between feature and a bug and it adds a new ini which is cleaner to do in the last supported version. Also there's a work around so I would suggest to use that in 8.0 and 8.1:

The solution when using the "php:fpm-alpine" image is to set "sendmail_path" to: '/usr/bin/dos2unix -u|/usr/sbin/sendmail -t -i'

[aly9]

Dears, Is there any final class for sending emails to be used in PHP version greater than 8 without changing the sendmail_path ? Since we are not allowed to change the .ini file configuration in some shared hosting.

[bukka]

@aly9 no but you should contact the hosting provider to change it globally in their configs.

[djixas]

I'm experiencing this issue too and it's driving me insane , both on PHP 8,1 and 8,0, and i am not using alpine

https://github.com/Shardj/zf1-future/issues/274

[aly9]

@bukka Thanks for your reply, but I have modify my sending emails script to use PHPMailer Class and it is working fine.

[zaphod77]

Sendmail binaries on linux, and pretty much every other OS using LF only are designed to parse normal text for the operating system when piped to directly.

They internally convert to match the RFC after, which causes a doubled up CR usually with php 8.x now.

Actual Sendmail does have a -ba parameter, which does place it into ARPANET mode, which will do the right thing when presented with a text file containing CRLFs for injection. specifying the -ba option in sendmail path does fix the issue, but naturally, the email tester doesn't do this. So the email test fails.

qmail's fake sendmail binary does NOT SUPPORT -ba. not even the alternate usendmail wrapper does. postfix, and most other MTAs that provide a fake sendmail binary do not support ARPANET mode like a real sendmail binary does. The only one that i've seen that does this is actual sendmail itself.

The sendmail binary has always accepted email for injection in the OS native text format. All other MTAs that have fake sendmail binaries do the same. They all still send RFC compliant emails, doing the needed conversion. When telnetting or s_clienting to the smtp port, telnet itself usually handles this part, and will convert the line endings properly.

I'm sorry, but PHP is wrong here. it's mail() function expects either smtp on windows, or a sendmail binary on linux/bsd/osx/etc. And sendmail has ALWAYS accepted messages in the UNIX (linefeed only) text scheme on unix, and it always will.

This isn't a silly RFC issue. Sendmail is NOT breaking the RFC by accepting and interpreting piped data stored in native OS text format. the RFC is for message transmission, and when using sendmail binary, php is not transmitting itself. it's handing off to the sendmail binary.

Exim is one that's smart enough to just do the right thing. It actually just strips out the \rs and then puts them back when it send the message :). I'm sure there are some others, but as we can see, it's just not widespread enough.

Now this IS something that configure should be able to catch.

actual sendmail will respond to "sendmail -ba" with 550 Recipient names must be specified

in this case, it's real sendmail, and -ba should be added to the options automatically.

Any other response mean sit's not actual sendmail, we can't do that.