On Windows in native mode detect proper color handling

[maxandersen]

Description In #1029 we avoided the windows ansi mess by disabling it by default on windows and not in known detectable ansi terminal. The only reason we didn't handle it better is that to do so we need to use native apis.

native apis to differentiate between wether content is being piped or not. Java cannot do that alone. native apis to use setConsoleMode to activate the ansi capable feature that is available in Windows 10 since ~2017 but for unknown reasons not enabled by default.

Implementation ideas Based on notes from @dmlloyd at https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/windows.20console.20logging/near/176014057

Highlights: "org.graalvm.nativeimage.ImageInfo#inImageRuntimeCode will tell you if you're in an image"

"basically you start by creating a class annotate it with a @Platforms(Platform.WINDOWS.class)"

"you'll need to also create a class similar to this one: https://github.com/oracle/graal/blob/master/substratevm/src/com.oracle.svm.core.windows/src/com/oracle/svm/core/windows/headers/WindowsDirectives.java"

"it should only include the header file that SetConsoleMode is in then basically all you do is something like this (which might be wrong):

@CFunction
public static native boolean SetConsoleMode(UnsignedWord handle, UnsignedWord mode);

" Then fiddle a bit until it works.

[maxandersen]

After lots of fun figuring out how to build native-image on windows I hit https://github.com/oracle/graal/issues/1675 making it impossible to know what errors I got in native code on windows. So putting this on hold until found a way around that (probably need new graalvm release)

[remkop]

@maxandersen An alternative approach to calling the setConsoleMode native API may be to use Jansi and picocli-jansi-graalvm.

In native images Jansi has issues. picocli-jansi-graalvm works around these issues.

Enabling it works similar as with Jansi:

import picocli.jansi.graalvm.AnsiConsole; // not org.fusesource.jansi.AnsiConsole
// ...
public static void main(String[] args) {
    boolean windows = System.getProperty("os.name").toLowerCase().startsWith("win");
    if (windows) { AnsiConsole.systemInstall(); } // enable colors on Windows
    // now do cool stuff with ANSI escape codes
    int exitCode = new CommandLine(new MyApp()).execute(args);
    if (windows) { AnsiConsole.systemUninstall(); } // cleanup when done
    System.exit(exitCode);
}

The advantages of using Jansi are:

• Jansi works in Windows 10 and older versions of Windows.
• The application does not depend on registry settings or external applications, it can safely emit ANSI escape codes
• Jansi is clever enough to strip out ANSI escape codes when not in a TTY interactive terminal. This is good because you don't want the escape codes to end up in a log file when the application output is redirected to a log.

Generally, be aware that setConsoleMode alone is not enough. It is especially important to not emit ANSI escape codes when the output is redirected. In pure Java the closest approximation for detecting this is to check if System.console() returns null. Jansi uses a native method that is more reliable.

For reference, picocli uses a set of heuristics to determine whether to emit ANSI escape codes or not. These may be useful for Quarkus as well.

[dmlloyd]

Using a native method from a native image is OK, but we don't want to rely on native methods (or more specifically, we don't want to rely on having to distribute platform-specific libraries) when we're in a pure-Java environment.

[maxandersen]

@remkop as @dmlloyd points out relying on jansi have issues on its own but this issue wouldn't go away in native - and once we are in native mode jansi would be superfluous as far as I can see.

[dmlloyd]

That said, picocli's heuristics are superior to our own.

[remkop]

Using a native method from a native image is OK, but we don't want to rely on native methods (or more specifically, we don't want to rely on having to distribute platform-specific libraries) when we're in a pure-Java environment.

Understood. Please be aware that this means no ANSI colors outside of native images.

That said, picocli's heuristics are superior to our own.

Actually, the discussion above made me realize there’s currently no way to tell picocli to always emit ANSI escape codes when in a TTY (interactive console). This would be needed when applications enable colors with the SetConsoleMode without Jansi. I’ll try to fix that soon.