Wrong CRLF translation when running external commands

[dmik]

This simple test demonstrates it.

TestCRLF.sh:

echo "STDOUT Line 1"
echo "STDOUT Line 2"
echo "STDOUT Line 3"
>&2 echo "STDERR Line 1"
>&2 echo "STDERR Line 2"
>&2 echo "STDERR Line 3"

TestCRLF.pm:

# This will just spit STDOUT and STDERR to console
system("./TestCRLF.sh");
# This will store STDOUT to $out
my $out = `./TestCRLF.sh`;
print "[$out]\n";

Expected output:

STDOUT Line 1
STDOUT Line 2
STDOUT Line 3
STDERR Line 1
STDERR Line 2
STDERR Line 3
STDERR Line 1
STDERR Line 2
STDERR Line 3
[STDOUT Line 1
STDOUT Line 2
STDOUT Line 3
]

Actual output:

STDOUT Line 1
             STDOUT Line 2
                          STDOUT Line 3
                                       STDERR Line 1
                                                    STDERR Line 2
                                                                 STDERR Line 3
                                                                              STDERR Line 1
                                                                                           STDERR Line 2
                                                                                                        STDERR Line 3
                                                                                                                     [STDOUT Line 1
STDOUT Line 2
STDOUT Line 3
]

This is a very annoying behavior which can be seen in many *nix perl scripts (e.g. in git ones, see https://github.com/bitwiseworks/git-os2/issues/2).

[dmik]

Note that it's irrelevant if the external program (sh.exe in the above case) spits CRLF or just LF — the result is the same. Looks like Perl a) unconditionally converts CRLF to just LF (\n in C) when reading command output and then does not convert LF back to CRLF when writing to its own console. This behavior doesn't seem to be affected by binmode(STDOUT, ":raw :crlf") and similar settings in any way.

It's actually quite questionable how it should behave. My bet is that binmode settings should be respected and if it's :crlf (which is default on OS/2 IIRC) then CRLF should be translated to LF when taking output from external commands back to CRLF when flushing it to the console.

[dmik]

There is also another consequence of this mess and it's also in git, see https://github.com/bitwiseworks/git-os2/issues/2#issuecomment-797743840. In short, when piping Perl currently always sends EOLs as CRLF to other programs and this breaks git apply which doesn't expect CRLF on input (it results in to an extra CR (0x0D) character at the end of the line) which prevents it from matching strings in patch files which makes it fail to apply patches containing CRLF.

Also, as it can be implied from https://perldoc.perl.org/functions/binmode, the default mode for files is TEXT in Perl. And files that need no translation must be specifically put into binary mode with binmode. This is what git-add--interactive actually does. So, from what I see, we should really just fix Perl to fully respect the binmode setting when piping and it should fix all issues related to EOLs.

[dmik]

I'm not entirely right in the comments above, it's really complicated and mixed up (as always with this ugly EOL legacy). git-add--interactive does not set binary mode on OS/2. It remains in text mode. This makes things really complex there: when piping output of an external command to its own console, perl does never do LF to CRLF translation (ignoring binmode setting completely). However, when piping its own output to the external command's input, it always does LF to CRLF translation (thereby also completely ignoring binmode).

So, first, we should make Perl respect binmode when piping. This will fix EOLs when piping command's output to the console (see the description of this ticket) but will NOT fix the git apply problem. For that, we will then have to fix either git-add--interactive to switch to binary mode when piping the patch to git apply or fix git itself to correctly handle CRLF in patch files, regardless of Perl. Given that git-add--interactive does not contain any special hacks in this regard for Windows (where there is supposedly no problem with this script), I suppose that in Windows the latter is applied (i.e. CRLF is accepted in patches). We will see what's best for us after we fix Perl.

[dmik]

Some observations. Setting binmode to raw (i.e. binmode($fh, ':raw'); or just binmode $fh;) before the print statement writing the patch file to the piped git apply command in git-add--interactive fixes the problem because Perl stops translating LF to CRLF and just puts what the patch has. So, perhaps nothing should be done in that regard in Perl itself — as print in question directs its output to the "file handle" associated with the pipe command" rather than to STDIO, binmode for STDIO should not actually be obeyed — all platforms should simply use binmode in that case (as per Perl docs).

This, however, doesn't fix the output from the external command. There is no "file handle" to reset to binary and therefore it remains bound to the Perl's own console and should respect its binmode setting. What fixes output is changing the default PerlIO layer to raw (by doing set PERLIO=raw in the environment) but this OTOH breaks input to git (and I guess many other things too) because Perl continues to translate LF to CRLF there — despite the raw mode set in PERLIO and even the direct (binmode($fh, ':raw') call. Weird...