is it defined/documented which newlines are used by Write-Host and Out-File

[philcerf]

Prerequisites

• [X] Write a descriptive title.
• [X] Make sure you are able to repro it on the latest released version
• [X] Search the existing issues.
• [X] Refer to the FAQ.
• [X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

Hey.

I have some trouble finding documentation on how e.g. Write-Host or Out-File do newlines, in particular whether they do CRLF or LF newlines.

What I do is:

• Process some stuff in PowerShell (e.g. a command like Get-Process | ..., execute that via PowerShell’s -Command option and capture the output of the whole things for further processing.
• Or, also executing something in PowerShell, but permanently storing it in a file via ... | Out-File ....

As far as I understand (please correct me if wrong), Out-File just takes what it gets and makes not conversion (regardless of whether it's executed under Windows, or Linux, or WSL or Cygwin). So it actually depends on what I pipe into Out-File whether I get CRLF or LF, right?
For example, ConvertTo-Json -Compress seems to give me no newlines at all, makes sense, while ConvertTo-Json XML seems to give me CRLF, at least under Cygwin/Windows, and there seems no direct way to change that. (Though I can live with it, if it's at least deterministic).

However, with Write-Host things are different.

• First, I use it, because without, PowerShell seems to do all kinds of strange modifications to the output, replacing text by ... or truncating it at some width.
• Second, when using -NoNewline, it seems Write-Host doesn't change anything, it adds no newlines (neither CRLF nor LF) and if there are multiple objects, they're merely concatenated without separator.
• But without -NoNewline, things seem to differ between versions. With PowerShell 5.1, when executed from Cygwin or cmd.exe I get LF newlines, but with 7.x I get CRLF.

Especially that last point is IMO problematic, when one wants to write code that works with future versions.
Is it now going to stay forever CRLF or could that change again in a future version, perhaps even depending on the current platform?

It seems to be nowhere documented what it actually uses as newline and on which platform, neither seems there to be any mention in https://learn.microsoft.com/en-us/powershell/scripting/whats-new/differences-from-windows-powershell?view=powershell-7.4 about that change.

Also, doing e.g. ... | Write-Host -Separator "n"` seems to be ignored for unknown reasons. :-( That would of course be the best way to get some consistent behaviour.

Thanks, Philippe.

Expected behavior

Consistent (and especially backwards compatible) behaviour of whether the cmdlets use CRLF or LF.

Or at least some definite documentation on what they do (on which platform).

Also, would be nice if `-Separator` works also for piped input?

Actual behavior

Spurious change from LF to CRLF between Windows PowerShell 5.1 and newer versions. Unclear status if there will be further changes (which might then break things).

Error details

No response

Environment data

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.19041.4412
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.4412
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

and

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.19045
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[237dmitry]

I think [System.Environment]::NewLine

[philcerf]

@237dmitry

I don't think that can be it, because under 5.1, where I see LF-newlines I get:

[System.Environment]::NewLine | Format-Hex

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   0D 0A                                            ..

While on 7.x, where I see CRLF, I get:

[System.Environment]::NewLine | Format-Hex

   Label: String (System.String) <2FCD9D69>

          Offset Bytes                                           Ascii
                 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
          ------ ----------------------------------------------- -----
0000000000000000 0D 0A                                           ��

[philcerf]

IOW, that value claims in both cases it would be CRLF, while in the old versions I see LF newlines.

[rhubarb-geek-nz]

Yes, Write-Host on 5.1 does indeed give LF, and does differ from what Write-Output gives you.

D:\>PowerShell Write-Host 'foo' > out

D:\>PowerShell Format-Hex -path out

           Path: D:\out

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   66 6F 6F 0A                                      foo.

The .NET Framework WriteLine does give you CRLF so it is a PowerShell 5.1 oddity

PowerShell "[System.Console]::WriteLine('foo')"

Note that generally separator refers to something between records, not after every record so would not work for line endings.

Out-String also gives you CRLF on both 5.1 and 7.* on Windows

PS> ('foo' | Out-String).Length
5

I would suggest that this project is most concerned with 7.* and it is more consistent regarding using [Environment].NewLine

[philcerf]

I would suggest that this project is most concerned with 7.* and it is more consistent regarding using [Environment].NewLine

Okay, so does that mean (which I guess was my original request for documentation), that in current and future PowerShell versions, newline is and will always be CRLF respectively always whatever [Environment].NewLine says (which then may be different on different platforms)?

I think this should go in the docs (just like e.g. Python docs mention for print what the newline will be and how one can set it), or at least in that differences between 5.x and beyond.

Note that generally separator refers to something between records, not after every record so would not work for line endings.

It would be okay for me, if -Separator would just work between, as I could simply ignore the final (and only) newline.
But even that doesn't seem to work, at last not when piping.

$ powershell -Command '$a = ("foo", "bar"); $a | Write-Host -Separator "_"' | xxd 
00000000: 666f 6f0a 6261 720a                      foo.bar.
$ powershell -Command '$a = ("foo", "bar"); Write-Host -Object $a -Separator "_"' | xxd 
00000000: 666f 6f5f 6261 720a                      foo_bar.

(the same with 7.4, just that that uses CRLF, rather than LF).

Also, is there a way to set/change [System.Environment]::NewLine? The property itself is read-only and a quick search in google didn't bring up anything? So can I get current PowerShell to e.g. always use CRLF or always LF, regardless of the platform?

Thanks, Philippe

[rhubarb-geek-nz]

So can I get current PowerShell to e.g. always use CRLF or always LF, regardless of the platform?

System.Environment.NewLine is provided by the .NET runtime and is dependent on the version of OS that is running, eg it is correct for the accepted conventions on that platform. PowerShell simply uses that.

If you have a file format which specifies using CRLF then I suggest it is the responsibility of the formatter/converter for that format to use the correct line ending, eg the HTTP protocol uses CRLF independent of the platform for separating headers etc, but canonical XML uses just LF. If you have git or subversion so configured it can do the line ending conversions so it is correct for text files on which ever system you are working on.

I would recommend not fighting the system with a global solution and treat the line ending problems as localised problems for a specific situation. If you are using Cygwin and using windows files and programs then you are simply asking for problems trying to pretend that a Windows system is a POSIX system.

If you are using WSL then run a Linux PowerShell within WSL, and when in Windows environment, use the Windows version. I turned off file sharing and program interoperation between Windows and WSL because it was far more trouble than it was worth.

[rhubarb-geek-nz]

On the difference between Write-Output and Write-Host on PowerShell 5.1, for data output that is intended to become part of the input to another pipeline, or output to a file, then use Write-Output. If it is supposed to be for the user to see on the console, then use Write-Host. In that case, if Write-Host is always going to be printed on the console then the difference between LF and CRLF makes no difference because it just appears on the screen and is not saved anywhere.

[mklement0]

Let me try to summarize:

• PowerShell generally uses the value of [Environment]::NewLine, i.e. the platform appropriate newline sequence or character when writing data, but accepts either form when reading data (and even accepts CR-only(!) newlines).

• With respect to writing, there are a few exceptions:

  • In the legacy edition, Windows PowerShell, Write-Host when its output is sent to stdout in a CLI call inexplicably uses LF-only newlines; this should be considered a bug, and PowerShell 7 fortunately exhibits the correct, platform-appropriate behavior; as noted, Write-Host isn't meant for outputting data; that is what Write-Output / implicit output are for, where the CLI exhibits the expected behavior even in Windows PowerShell.

  • If you use Write-Host -Separator ([Environment]::Newline) - which fundamentally only works on direct arguments rather than pipeline input - the separator is only placed between multiple arguments, and in CLI use the implicit trailing newline will still be LF-only in Windows PowerShell.

  • Here-strings:

  • When you create them interactively (which isn't common), they curiously always use LF-only newlines, in both editions, on all supported platforms.

  • In script files, they use the same newline format as the enclosing file, irrespective of platform; e.g., if a script file has CRLF newlines, any here-strings created in them will have CRLF newlines too.

• Specifying the newline format explicitly when writing to files:

  • The following feature request was declined, unfortunately: #3855
  • Therefore, creating a single, multi-line string with the desired newline sequence (including a terminating one) and saving it with Set-Content -NoNewLine or Out-File -NoNewLine are the only (PowerShell-native) options.

• Problematic use of newlines / platform-appropriate newlines:

  • When you send a string to the pipeline to an external program, a platform-appropriate newline is invariably appended, which is problematic in two cases:

    • 5974

    • 13579

  • 14444