Support for interactive console application on Windows

[ppastercik]

Description

When developing a console PHP application that works with user input (for example implemented using symfony/console or laravel/prompts), interactive work using console input (for example using arrow keys), is available on Linux environments but is missing on Windows environments. This limits the development of console-based cross-platform PHP applications. It is circumvented either by limiting interactivity (which is the case with symfony/console) or by using a fallback to a non-interactive solution (which is the case with laravel/prompts, which has a fallback to symfony/console).

Specifically, this can be seen, for example, in the list selection. When on Linux we can select values using the up and down arrows. Alternatively, when typing a password (invisible input), on Windows we need to use a special program to get the input without displaying it in the console, see hiddeninput.exe for symfony/console.

Here is a sample of the interactive application control, which is now not possible on Windows:

On Windows OS this is due to the fact that it is now not possible to update the console input stream mode from user code.

For interactivity, we need to allow developers to set these console modes:

Mode	Purpose	Default	Needed
ENABLE_ECHO_INPUT	Writes input to the sceen buffer	Enabled	Disabled
ENABLE_LINE_INPUT	Waits for carriage return	Enabled	Disabled
ENABLE_PROCESSED_INPUT	System processed Ctrl+C instead of input buffer	Enabled	Disabled
ENABLE_VIRTUAL_TERMINAL_INPUT	Converts input into sequences (for arrow keys, backspace, etc.)	Disabled	Enabled

Proposal

Adding support functions to PHP that allow setting the mode (using the SetConsoleMode function) for the input console stream without having to enable any PHP extension.

The question is the form of these functions.

We can use separate support functions (inspired by sapi_windows_vt100_support function), see the sample user code:

<?php
    $defaultEcho = sapi_windows_echo_input_support(STDIN);
    $defaultLine = sapi_windows_line_input_support(STDIN);
    $defaultProcessed = sapi_windows_processed_input_support(STDIN);
    $defaultVt100 = sapi_windows_vt100_input_support(STDIN);

    sapi_windows_echo_input_support(STDIN, false);
    sapi_windows_line_input_support(STDIN, false);
    sapi_windows_processed_input_support(STDIN, false);
    sapi_windows_vt100_input_support(STDIN, true);

    // Now it reads by characters, not echo input, read "CTRL+C" as character, with arrow keys, etc. as characters.
    $readedChar = fread(STDIN, 1024);

    sapi_windows_echo_input_support(STDIN, $defaultEcho);
    sapi_windows_line_input_support(STDIN, $defaultLine);
    sapi_windows_processed_input_support(STDIN, $defaultProcessed);
    sapi_windows_vt100_input_support(STDIN, $defaultVt100);

Or we can use universal support function with constants, see the sample user code:

<?php
    $defaultMode = sapi_windows_input_mode(STDIN);

    sapi_windows_input_mode(STDIN, $defaultMode & ~SAPI_WINDOWS_ECHO_INPUT_MODE & ~SAPI_WINDOWS_LINE_INPUT_MODE & ~SAPI_WINDOWS_PROCESSED_INPUT_MODE | SAPI_WINDOWS_VT100_INPUT_MODE);

    // Now it reads by characters, not echo input, read "CTRL+C" as character, with arrow keys, etc. as characters.
    $readedChar = fread(STDIN, 1024);

    sapi_windows_input_mode(STDIN, $defaultMode);

Another question here is whether the input stream STDIN needs to be defined, or if it could be omitted, since this can only be used for the console input stream anyway.

Proof of concept

I created a proof of concept using separate support functions.

The implementation of the supporting functions in PHP is here: https://github.com/ppastercik/php-src/tree/proof-of-concept-laravel-prompts-on-windows

Sample of an interactive console application supporting the Windows platform. It uses the laravel/prompts package, whose implementation is extended with new support functions: https://github.com/ppastercik/proof-of-concept-laravel-prompts-on-windows

Tested on the following terminals / emulators with php artisan app:test:

Name	Working	Differences
CMD	Yes
PowerShell	Yes
Cmder	Yes
ConEmu - CMD	Yes
ConEmu - Msys2	Yes
Console2	Yes
ConsoleZ	Yes
Git Bash	Yes
MinTTY	Yes
MobaXterm - SSH	Yes	(backspace is 0x08 and not 0x7F)
MobaXterm - Bash	Yes	(run with "winpty php artisan app:test")
PuTTY	Yes
ZOC8 Terminal	Yes

Alternatives

I have been exploring various alternatives within the laravel/prompts issue to see if there is any way to achieve interactive console behavior without interfering with the PHP source code.

External exe file for reading input - working

It is possible to create a support program that reads a character from the command line and returns it to the application (similar to hiddeninput.exe in symfony/console).

But there is the disadvantage that it would have to be run for each character read. Or it would have to be run as a separate process, getting the input through a pipe, and the process would have to be terminated properly at the appropriate time (with the need to prevent multiple processes from running concurrently, for example if another process were to be started for more input in a moment).

Another disadvantage is that in addition to the PHP code, the corresponding exe file would have to be distributed (which can be a problem with antiviruses, etc.).

Using the FFI extension - working

It is also possible to implement this using the FFI extension, where the appropriate mode is set using the SetConsoleMode function. But with this implementation, developers need to enable the FFI extension.

Alternative to Linux stty - not working

Next, I explored the possibility of creating an alternative to Linux stty for Windows. But here I encountered the problem that even though the documentation of Windows console modes states that the mode can be transferred between processes (using same STDIN), in real testing some modes were resetting when starting or ending subprocesses (running within user code) that had the STDIN of a PHP process as input. Thus, this option turned out to be not working.

Possible problems

Returned line ending

The Linux variant using stty returns the line break as a \n character, while the modified Windows variant returns the line break as a \r character. However, this can be resolved in user code.,

Unsupported mode combination

There is a restriction that it is not possible to have the ENABLE_ECHO_INPUT mode enabled with the ENABLE_LINE_INPUT mode disabled (see ENABLE_ECHO_INPUT mode description in SetConsoleMode function). So when we choose separate support function implementation, then when both modes are off, and we want to turn them both on, we need to call the functions in the correct order. Or, if we don't know the default state and the target state is defined by variables, we call one of the functions twice (sapi_windows_echo_input_support, sapi_windows_line_input_support, sapi_windows_echo_input_support).

Codepage of console input stream

Error in the default console input stream codepage in the Windows environment. When setted to UTF-8 (which is the default setting in PHP), the internal PHP implementation for reading data from the console does not read the characters ěščřžýáíé etc. This is due to the read function reading the console stream, which returns a buffer containing test\0\0test for the string testěštest.

Thus, we need to set an 'oem' codepage before reading, like in symfony/console, which will allow those characters to be read as well. But then it needs to be undone, because if the user code wanted to output something between reading each character, it would have the wrong coding. Thus, to read each character, the codepage needs to be reconfigured each time.

To avoid setting the codepage to read every character, it would be useful to consider extending the sapi_windows_cp_set function to allow the codepage to be set only for STDIN so that the output could still be in UTF-8.

Reseting modes after termination of a subprocess (with inherited STDIN)

When a subprocess is started using the current STDIN as input, some modes will be reset after the subprocess ends. Specifically, the following modes will be reset: ENABLE_ECHO_INPUT, ENABLE_LINE_INPUT, ENABLE_PROCESSED_INPUT. However, the setting of the ENABLE_VIRTUAL_TERMINAL_INPUT mode will be preserved. Thus, it would be advisable to point this out in the documentation that in such a situation, the given modes need to be reconfigured.

Breaking change

None. Only adds new functions.

Operating System

Windows

Conclusion

If there is interest in adding these support functions, the question is the form of implementation. And another question is whether to extend the implementation of the sapi_windows_cp_set function to include the ability to set the codepage to STDIN only (which is mentioned in the "Codepage of console input stream" section under "Possible problems").

Is there any next steps I can do?

[nielsdos]

Very nice proposal and very well written. The way forward I think is proposing an RFC to the internals mailing list as it's about adding new APIs to the language. I'm sure some people will have opinions about the design. We generally use the mailing list instead of GitHub for such discussions. This is so that we can have a discussion on the mailing list with both maintainers and community members, i.e. having a greater reach. Please see https://wiki.php.net/rfc/howto for a how-to guide for an RFC.

It's also very good that you already have an implementation, it helps if people can have a look at some implementation.

[thg2k]

It would be awesome to have terminal control abstraction in core PHP. I usually get away with use system("stty ..."); but that only works on Linux of course. Big +1

[github-actions[bot]]

There has not been any recent activity in this feature request. It will automatically be closed in 14 days if no further action is taken. Please see https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea to understand why we auto-close stale feature requests.

[Wirone]

Was RFC proposed yet?

[ppastercik]

@Wirone Not yet, sorry.

In October I wrote an initial description of the proposal for the official PHP mailing list, as a first step before creating the RFC. See: Proposal: Add support for interactive console application on Windows

But then I had other priority things to deal with. Maybe I should have let you know here that I'll come back to it later.

I suppose this month I could get back to it and create an RFC.

However, before creating the RFC, I want to explore the possibilities of some abstraction over the console input, as @thg2k writes above (perhaps by calling the appropriate system functions that are otherwise called by the stty command). Because I'd like to have uniform control of console input independent of platform. But I want to see if this can be done easily, or if it would overcomplicate the RFC and thus delay the eventual implementation.

Plus I still want to explore more in depth working with the codepage when processing console input on Windows. Because my initial attempt showed that it is not possible to read UTF-8 correctly on Windows:

Error in the default console input stream codepage in the Windows environment. When setted to UTF-8 (which is the default setting in PHP), the internal PHP implementation for reading data from the console does not read the characters ěščřžýáíé etc.

So I would like to look into the possibilities, if it would be possible to directly modify the reading of the console input stream on Windows to support UTF-8 codepage correctly.

[github-actions[bot]]

There has not been any recent activity in this feature request. It will automatically be closed in 14 days if no further action is taken. Please see https://github.com/probot/stale#is-closing-stale-issues-really-a-good-idea to understand why we auto-close stale feature requests.

[sgolemon]

@auroraeosrose or @petk This proposal came up on PHP Roundtable last nigh and it seems like such a trivial thing to get moving (well maybe not trivial but not terribly contentious), but I'd end up having to spend the next week getting a windows build env set back up. I'm wondering if you have time (or know a windows pro who's looking for commit points) to help build out a more concrete version of this proposal.

Part of me wants to take it in the direction of a standalone extension which can be iterated on (there's a LOT in the windows-specific APIs we could explore beyond just these console functions). I've long had thoughts about doing something with macOS's CoreFoundation frameworks in similar ways...

Anyway, I wanted to resurrect this issue and see if we can get traction. If neither of you (or your minions) are available, I'll work on getting that windows build env going and see what I can do.

[auroraeosrose]

Already did a version awhile ago for myself that wraps gobs of windows API functions for 5.4 (ages ago) was starting an 8.x port but got sidetracked with life again. But yes this probably would be best fixed with some win32 api extension(s)

The question would be scope honestly. How much would you want in an extension? Does this belong in core or as a PECL/external project?

[sgolemon]

The question would be scope honestly. How much would you want in an extension?

I would say anything someone has raised their hand to say "that looks useful and s uniquely windows" and items closely related to that. Throwing in the kitchen sink might be a bit much. Like, I don't think we need an HTTPClient wrapper since we already have one of those that's OS agnostic. I also think that GUI APIs would be... a lot to try to make default. Something like that would be external dll at best.

I've also reopened the mailing list thread to get thoughts from there

Does this belong in core or as a PECL/external project?

If I'm honest.... probably PECL/external if we're going to go even a little bit "full send". If we keep scope tight, then I think it's reasonable to widen the scope of /win32 and make an honest builtin extension out of it.

[auroraeosrose]

A win32 with a very tight scope sounds like a great idea. However I just know what "tight scope" always ends up being. I suppose a really good RFC might alleviate some of that? And then leave the rest for PECL but allow the hooks external extensions might need (getting pointers out of structs etc, c apis to touch internal things without breaking the world)

But even if you tighten the scope down to just process stuff there's a LOT available ;) https://learn.microsoft.com/en-us/windows/win32/procthread/process-and-thread-functions yeah - that's just process stuff. Not to mention codepage support which absolutely would need some stuff exposed. Using UTF8 with windows apis is ... a fun adventure (it can be done but you have to like a bit of pain)

https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page (here's some of the pain - I had to hack the windows build system for PHP to get the manifest the way I needed)

Then there's also the whole fun of the way that we're switching between doing main and winmain for the windows specific cli...

It also really depends on what "specific" win32 stuff we have now in PHP (I've honestly never cataloged it all), what we might want to do in addition, and what internal stuff (look at win32/codepage) we might want to expose/allow users to touch but at the very least some hooks for extensions to do some manipulation on process/sapi/codepage stuff would be useful.