PowerShell output get corrupted/duplicated

[adamkbmc]

I've found that if you produce a lot of output from a PowerShell script, then the output get corrupted with duplicate output produced. I can reliably reproduce this using various Windows instances.

Attached is some example output from a Windows Server 2019 instance running agent version 3.1.501.0. I captured this by running "aws ssm start-session" from within a "script" session a Linux box. Once connected, I pasted in the following PowerShell loop which just produces 10000 lines of output:

for (($i = 0); $i -lt 10000; $i++) {
    "This is line {0}" -f $i
}

In the attached corrupted_output.txt file you see the end of 10000 output lines at line 10067. This is then followed by the PowerShell prompt .. but then the prompt gets repeated (a couple of times) followed by the last 38 lines of output again (lines 9962 to 9999). If you run the loop again you get the same result (but not always from line 9962, I've seen 9963 and 9964 as well)

corrupted_output.txt

[yuting-fan]

What is the terminal size (rows and columns) of the "Linux box" you used to send start-session requests?

[adamkbmc]

I think the terminal was 50 rows x 80 cols at the time but I don't see how that matters. I've run it with various terminal sizes with the same effect. This would seem to be a problem with how the hidden console window is being handled, specifically the console buffer size rather than the window size.

[yuting-fan]

Hello, I have tried using Session Manager CLI or Session Manager console, with a combination of bigger/smaller buffer size on the remote windows machine, and I'm not able to reproduce this issue. Can you please enable the debug-level logs on SSM agent, and share the SSM agent logs with us? You could DM me if it's a large file, or please feel free to open a support case to track it along.

[adamkbmc]

Did you just watch the terminal output or capture it? If you just look at the terminal buffer then you don't see a problem, as the control codes cause the cursor to shift and the duplicated output overwrites the previous output and it all looks fine. If you capture the stream (i.e. using "script") then you see the duplication.

For a normal user interactive session things seem ok but if you try to use the SSM session for any kind of automation, say running the session via a pipe, the output stream contains the duplicated data.

[yuting-fan]

As you called out already, the terminal does not return duplicate output because the control codes would cause the cursor to shift up hence overwrite the previous output. Session Manager aims to provide customers a shell-like experience with regard to sending desired input and displaying the correct output in a shell-like terminal. Capturing the output from the remote machine and redircting it to a local file line by line (which is what script command does) is a customization that Session Manager does not support.

To share more technical details, the output from Powershell can contain mix of human readable characters and control sequences like color mode, cursor position, overwriting of terminal, etc, and can be best interpreted by terminal emulator only. Additionally, the output can vary among different versions of Powershell or Windows machines. In case of the issue you reported, the pseudo terminal of Powershell could send output containing existing and new lines to represent entire terminal screen along with cursor positions to overwrite existing lines. This results in duplicate data when a custom client reads the output sequentially but looks as expected on terminal since duplicate data gets overwritten.

Please let us know if you have additional questions.