become a thing on windows

[dankamongmen]

In a Kantian sense, it would seem a good thing if notcurses worked on windows, for whatever definition of works that means. I very much doubt that I will ever undertake this work, but if some brave soul wishes to take a stab at it, or report problems, or whatever, here's a bug in which you might do that.

[dankamongmen]

https://github.com/microsoft/terminal would be our target, i am thinking

[sandorex]

I am interested at taking a stab at this, cannot promise anything though

Any pointers?

[dankamongmen]

I am interested at taking a stab at this, cannot promise anything though Any pointers?

Neat! Feel free to hit me up for any assistance you might need. Unfortunately, I don't have much advice at this moment -- I haven't written code for Windows in over twenty years. Probably the first thing would be prepping our build infrastructure to run on a Windows machine with VS/VC++ --- I chose CMake largely because it supposedly makes Windows development easier.

I doubt very much that we'll build out of the gate, but once we have some kind of "I run this, expect this, but instead get this" loop, we ought be able to advance it quickly.

Next would be determining what the state of terminal support is on Windows. Is terminfo a meaningful concept there? What are the major terminals? In what ways are they similar and different from UNIX terminals?

So there are a few questions of basic groundbreaking that still need answers IMHO.

[grendello]

Since it's a console library, can I suggest that we make sure it builds under mingw-w64 as well? It's easy to set up and it would be nice to be able to build and run the tests on one machine (with WINE). CMake is shipped with VS2019, so yes, it would make stuff easier, however another possibility is to use clang on Windows since with that you don't need the whole IDE installed, just the Windows SDK. With CMake all of MSVC, mingw and clang can be supported. In Xamarin.Android we use cmake to build our Windows bits on Linux/macOS with mingw-w64, here are 32-bit and 64-bit CMake toolchain definitions for this.

Like Nick, I haven't coded for Windows for more than 20 years, so I don't know the state of the console, but I suspect ncurses is not a big thing there and so terminfo might be hard to support, however there's https://github.com/mauke/unibilium/ which provides the necessary terminfo support, they also mention a thing or two about Windows. @dankamongmen maybe it would be good to consider Unibilium as the ncurses replacement for notcurses? Last time I tried to build notcurses on macOS, it couldn't find libtinfo (neither the shipped nor the homebrew ncurses had it - they may had the stuff compiled into the main library, I didn't check tbh) and so if we used Unibilium we'd be better off on systems where ncurses isn't native or just not well supported.

Lastly, @sandorex you might find https://github.com/Microsoft/vcpkg useful for building stuff on Windows as far as dependencies are concerned, here's how our LibZipSharp uses it.

[sandorex]

Some information I've gathered about current state of console on Windows (take it with a grain of salt)

Both the new and the old terminals should be treated as xterm-256color (this may change in the future) as said here

For all intents and purposes, the Windows Terminal should be treated as xterm-256color, and any bugs that might arise from that should be filed here, so we can close any remaining gaps.

Both terminals support truecolor (but ill have to test that)

[sandorex]

@grendello It seems unibilium is unmaintained so it would be better to use the neovim fork

[grendello]

@sandorex sounds good!

[sandorex]

I've started making a unibilium replacement, its made in c++ but will have c wrappers, would that be acceptable to be used?

Other question i have is, are there any uses for ncurses other than getting the capabilities ? Im adding extended terminfo support so it should be able to read anything ncurses can

[Charadon]

Has anyone tried compiling notcurses with msys2/cygwin? If not, I can give it a go, and report my findings.

[Charadon]

So, after attempting to compile it with msys2 and cygwin, it gets stuck at the same place OSX does, it can't find uchar.h. I feel like a workaround for this might be to add a compile-time option to disable unicode and use plain ASCII or use a more portable unicode library.

[dankamongmen]

https://github.com/dankamongmen/notcurses/runs/1101716618?check_suite_focus=true hark! a github action appears!

[dankamongmen]

when we do this, we'll want to package up in vcpkg

[magiblot]

Let me share my knowledge about Windows consoles with you. This is what I learned while working on https://github.com/magiblot/tvision.

Is terminfo a meaningful concept there?

I would say it's not. https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences contains all you need to know.

What are the major terminals? In what ways are they similar and different from UNIX terminals?

During many years Conhost was the only relevant console on Windows. In Windows, the Console is a feature provided by the operating system. Every process can be associated to one Console, which is an instance of Conhost. Until Conhost received support for virtual terminal sequences, native TUI applications could only interact with this console using the Win32 Console API instead of stdio. In order to create your own terminal emulator, you had to wrap it around Conhost in some inefficient way.

Windows did not have an equivalent of forkpty until 2018, when the Pseudo Console API was introduced. This technology not only provides a PTY infrastructure, but is also a translation layer for client applications using the legacy Win32 Console API. Windows Terminal is an example of a terminal emulator relying on this API.

The above means that Windows developers have not had much time to write terminal emulators with diverging capabilites. Therefore, terminfo has little importance here.

Unix-like environments like MSYS2 or Cygwin may use terminfo because most applications running in them are ported straight from Unix systems. But terminfo won't be of any use if your application is running under Conhost unless you assume TERM=xterm-256color, like someone said earlier.

So, after attempting to compile it with msys2 and cygwin, it gets stuck at the same place OSX does, it can't find uchar.h.

You won't face this issue in MinGW or VC++.

Windows, sadly, loves UTF16. It is likely that we would need to handle UTF16 from Windows applications. We should support emitting UTF16 as output, perhaps.

UTF-16 support is certainly not necessary. You can interact with the console in UTF-8 as long as you configure the console codepage properly. Additionally, since 2018, UTF-8 is also supported in the C Runtime. So you can write a fully capable Unicode application on Windows without a single wide char, and I find it a good idea to encourage users to do so. The problem in notcurses will be that all interfaces assuming wchar_t to be 32-bit (if there are any) will be ill-formed on Windows.

Another difference compared to Unix terminals are character widths. While double-width characters are supported as you would expect, combining characters are drawn as single-width characters. For example, in Windows Terminal, the sequence में (1 base character + 2 combining characters) is 3 columns wide, even though the glyph is rendered correctly.

Cheers.

[dankamongmen]

recent events have me hoping that this will see life by 3.0.0 this fall

[michaelsbradleyjr]

I was re-reading this discussion and doing a bit of research.

I stumbled across this: https://github.com/huangqinjin/wmain.

I'm not entirely certain what that project is trying to accomplish, but the information in the README seems like it might be relevant/helpful.

Also, I recently learned that MSYS2 now offers builds for ucrt64. Whether using ucrt64 would make things better/easier than using mingw64, I'm not sure, but it may be worth considering.

[dankamongmen]

@magiblot i had somehow missed your post full of wisdom and knowledge until now. thank you for all the info!!

[dankamongmen]

heeeeeeeeeeeeeey youuuuuuuuuuu guyyyyyyyyyyyyyyyys i've got some surprise news: i have a hacked-up Notcurses successfully building on Windows 10 and running in the current Windows Terminal.

i'm very much out on a limb with it; it absolutely will not be merged in its current state. but as a PoC, it's there. this can be made to work. many thanks to @michaelsbradleyjr, whose recent OSX work motivated me to complete the Triforce. i'll put some more details here. i think maybe two full days of eager, focused hacking will have it beaten into shape.

[dankamongmen]

@magiblot i'm sure you are, but are you familiar with @klamonte's Jexer? you both seem to share a fascination with mid90s Borland products (understandable; I too grew up on bootleg copies of Turbo Pascal and Turbo Assembler).

[magiblot]

Hey Nick, thanks for the acknowledgement. Yes, I know of Jexer (although I never used it as a programmer) and yes, Borland products fascinate me for some reason (although Turbo Vision is a couple of years older than me and it was never part of my childhood).

[dankamongmen]

During many years Conhost was the only relevant console on Windows. In Windows, the Console is a feature provided by the operating system. Every process can be associated to one Console, which is an instance of Conhost. Until Conhost received support for virtual terminal sequences, native TUI applications could only interact with this console using the Win32 Console API instead of stdio. In order to create your own terminal emulator, you had to wrap it around Conhost in some inefficient way.

not that it particularly matters, but this doesn't make any sense to me. couldn't you just write your own win32 gui application that basically implemented a pty (if not by that name/API), and then run a shell on it, and call it a day? sure, you might have to write some code to "attach" to whatever IPC interface you exported, but all a terminal is (to quote myself in the Notcurses book) is "two buffers and a line discipline", no? you get encoded chars in, you render them to your canvas, you get input events from os, you turn them into encoded crap for the output buffer. what role was the "service"? i understand you couldn't say log system messages (it's this sense that i usually assign to "consoles"), but the terminal emulation paradigm seems quite independent of any os service....no? @magiblot

[magiblot]

couldn't you just write your own win32 gui application that basically implemented a pty (if not by that name/API), and then run a shell on it, and call it a day?

Before the ConPTY API was introduced the answer to this question would be "no". You could make such a terminal emulator, but there would be almost no native applications that you could run in it. When an application uses the Win32 Console API it does not print a stream of characters through stdio (as is the case in Unix). Instead, it communicates directly with the console process (and you are not allowed to intercept this communication).

My guess is that the Windows Console's arquitecture was heavily influenced by the kind of applications it would end up running: text-based MS-DOS programs. That kind of applications used BIOS interrupts and direct memory access to change the cursor status and draw things. Thus, when the Windows Console was first designed probably nobody thought of the character-based streams that are the essence of Unix terminals.

[michaelsbradleyjr]

Random, but doesn't Admiral Grace Hopper look totally shopped in the picture in that Microsoft devblogs post?

Maybe it's the lighting with respect to her dark-colored uniform.

[magiblot]

This other blog post explains with greater detail the internal design of the Windows Console.

[dankamongmen]

Before the ConPTY API was introduced the answer to this question would be "no". You could make such a terminal emulator, but there would be almost no native applications that you could run in it. When an application uses the Win32 Console API it does not print a stream of characters through stdio (as is the case in Unix). Instead, it communicates directly with the console process (and you are not allowed to intercept this communication).

mother of god

[dankamongmen]

This other blog post explains with greater detail the internal design of the Windows Console.

oh yeah, @bitcrazed is a good man, and thorough

[dankamongmen]

we're getting awfully close using the MSYS2 method: https://github.com/dankamongmen/notcurses/runs/3136423360

once this is done, i'll bring in native VSC++ support, if it's worthwhile to do so.

[dankamongmen]

things we still need:

• a solution for acquiring EGC width
• termios equivalents
• signal equivalents / workaround
• replace poll() with WSAPoll()

[bitcrazed]

LOL :) Thanks for the kind words, @dankamongmen 😁

Interesting thread and super-cool project! How did I not know about this until now?

And LOVE the references above to Borland's products and TurboVision - I spent my uni years, and the first few years of my professional career building TurboVision (and then OWL) apps. Such a fun time! 😜

Glad you enjoyed my series of posts on terminals, and Windows Console. Alas, I moved on from the Terminal team to work on other stuff before I got to close the series with a post all about the new Windows Terminal. However, I guess our Terminal unveil presentation at Build 2019 provides most of the details, but in video format 😜

I'd encourage y'all to file issues on the Terminal repo if you run into issues - I am pretty sure Kayla, Dustin, and the rest of the Terminal team would be happy to help you figure out a solution.

[dankamongmen]

I'd encourage y'all to file issues on the Terminal repo if you run into issues - I am pretty sure Kayla, Dustin, and the rest of the Terminal team would be happy to help you figure out a solution.

heh, i actually became a MSFT coworker of yours almost exactly one year ago, where i punch 40gbps holes through azure for satellites to blast comms through, and try to avoid using or booting my windows machine as much as possible. hello niblack, L66 from the atlanta office. good to hear from you man! and if you've never seen it, get a good glass of scotch and watch this https://www.youtube.com/watch?v=dcjkezf1ARY @bitcrazed

[dankamongmen]

alright, just about there. i've stubbed out a lot of signals- and termios-related stuff that we'll need resolve. right now we'd link except for open_memstream(), which i'm thinking we can either:

(a) implement for windows, but we'd need tease apart their FILE*, which sounds very bad, so (b) #ifdef everywhere and have two different paths, unthinkable so (c) implement an isomorphism to open_memstream()+stdio using CreateFileMappingA() and macros , ick, maybe, or (d) implement our own mmap() + CreateFileMappingA()+stdio abstraction

the last could introduce MMAP_HUGETABLES and such, and be a net win. definitely not what i planned on doing tonight, though. let's take a look through those stacks and see how much printf() is being used vs puts().

[dankamongmen]

MSYS2-built notcurses-info =]

[ghost]

• a solution for acquiring EGC width

Would the "CSI 16 / 6t", "CSI 14 / 4t" et al get you that? If the terminal is close to xterm you can get row/col cell counts and x/y pixels.

(Just not sure what EGC means in this context.)

[o-sdn-o]

It makes sense to integrate with Windows by using adapters such as MSYS2/CYGWIN only if you plan to use notcurses in versions up to Windows 10 (XP/Vista/7/8), or if you want to use WinAPI inside TUI. Windows 10 is armed with the WSL2 subsystem and it works great. It allows you to directly run code compiled for Linux. WSL2 is a better and more natural environment, especially if you don't need WinAPI inside TUI.

sorry for machine translation

[michaelsbradleyjr]

MSYS2 has very different goals than Cygwin: https://www.msys2.org/docs/what-is-msys2/

MSYS2 focuses on building native software built against the Windows APIs.

By extension, a notcurses build with/for MSYS2 can be used (with static linking) to build native Windows executables that can downloaded and run on Windows computers that have nothing of MSYS2 installed (or Cywgin, for that matter).

See: https://github.com/dankamongmen/notcurses/issues/1956#issuecomment-883452380.

You may be aware of all the above already; if so, my apologies for telling you what you already know; but I've witnessed lots of confusion re: MSYS2.

Also, I think that building it in WSL2 is pretty great, and a build for Cygwin could be nice too. What's more, cmake is flexible enough to support build logic with the Microsoft toolchain in addition to MSYS2, et al. So none of these options need be mutually exclusive.

[o-sdn-o]

Related https://github.com/microsoft/Windows-Dev-Performance/issues/15

[michaelsbradleyjr]

Related microsoft/Windows-Dev-Performance#15

Yes, related to MSYS/Cygwin.

Note that MSYS2 is different: without the POSIX emulation layer, you shouldn't 🤞 get the same kind of performance problems.

Copy/pasting from my comment I linked to earlier:

MSYS2 is an independent rewrite of MSYS, based on modern Cygwin and MinGW-w64 with the aim of better interoperability with native Windows software.

[MSYS2's] shell and core tools exist mainly to allow porting Unix programs to run natively on Windows (i.e. without requiring a POSIX emulation layer).

There still could be some issues with MSYS2 MINGW64, but I’m guessing most are cleared up with MSYS2 UCRT64, though maybe one has to steer clear of pthread emulation… I’m not sure.

[WSLUser]

As stated in other issues, WSL2 should not be considered a Windows version of notcurses. That's still a Linux version. Yes you can run notcurses under WSL2 with Windows Terminal or other terminals like ConEmu or Mintty. This issue is about getting Windows native so there's no middle man (WSL2 in this case)

[dankamongmen]

our latest Windows Action runs up against:

D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Plane.cc.obj):Plane.cc:(.text$_ZN4ncpp5Plane9map_planeEP7ncplanePS0_+0x189): undefined reference to `__stack_chk_fail'
D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Plane.cc.obj):Plane.cc:(.rdata$.refptr.__stack_chk_guard[.refptr.__stack_chk_guard]+0x0): undefined reference to `__stack_chk_guard'
[ 38%] Building C object CMakeFiles/zalgo.dir/src/compat/compat.c.obj
D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Tablet.cc.obj):Tablet.cc:(.text$_ZN4ncpp8NcTablet10map_tabletEP8nctabletPNS_9NotCursesE+0x189): undefined reference to `__stack_chk_fail'

[dankamongmen]

our latest Windows Action runs up against:

D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Plane.cc.obj):Plane.cc:(.text$_ZN4ncpp5Plane9map_planeEP7ncplanePS0_+0x189): undefined reference to `__stack_chk_fail'
D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Plane.cc.obj):Plane.cc:(.rdata$.refptr.__stack_chk_guard[.refptr.__stack_chk_guard]+0x0): undefined reference to `__stack_chk_guard'
[ 38%] Building C object CMakeFiles/zalgo.dir/src/compat/compat.c.obj
D:/a/_temp/msys/msys64/ucrt64/lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: CMakeFiles/notcurses++.dir/objects.a(Tablet.cc.obj):Tablet.cc:(.text$_ZN4ncpp8NcTablet10map_tabletEP8nctabletPNS_9NotCursesE+0x189): undefined reference to `__stack_chk_fail'

this has been dealt with. next is, i think, pulling out the terminfo stuff on windows, where it doesn't seem to work.

[dankamongmen]

we now build all targets in MSYS2.

[michaelsbradleyjr]

we now build all targets in MSYS2.

🥊 🏋️‍♂️

[dankamongmen]

i don't see any of them running, but some at least link, at least when launched within the msys2 environment. as noted elsewhere (#2009), i imagine this is due to terminfo on windows being questionable. i'll work some on it tomorrow, probably. there are almost certainly other issues as well. but everything compiles.

[michaelsbradleyjr]

If you run from cmd.exe or PowerShell, it's likely C:\ucrt64\ucrt64\bin isn't in PATH.

I mentioned in PR #1978 that putting it in PATH didn't work, but earlier today I tried it again and it did work, so previously I must have fat fingered it or accidentally put : instead of ; when doing (in cmd.exe):

set PATH=C:\ucrt64\ucrt64\bin;%PATH%

Prior to your latest changes (noting that just to be accurate), when running notcurses-info.exe in an MSYS2 UCRT64 context in Windows Terminal, it runs but does nothing and the exit code is 1. Running it from cmd.exe or PowerShell in Windows Terminal, having set PATH appropriately, it runs but does nothing.

[dankamongmen]

Prior to your latest changes (noting that just to be accurate), when running notcurses-info.exe in an MSYS2 UCRT64 context in Windows Terminal, it runs but does nothing and the exit code is 1. Running it from cmd.exe or PowerShell in Windows Terminal, having set PATH correctly, it runs but does nothing.

is it doing something different now? roger-wilco regarding PATH

[michaelsbradleyjr]

I'm pulling commits now and will do a build, etc. Will give an update soon.

[michaelsbradleyjr]

is it doing something different now?

No, it does not. make now finishes without error at 100%. 🎉 But runtime behavior is the same.

When running notcurses-info.exe in an MSYS2 UCRT64 context in Windows Terminal, it runs but does nothing and the exit code is 1 (per echo $?).

Running it from cmd.exe or PowerShell in Windows Terminal, having set PATH appropriately, it runs but does nothing.

[dankamongmen]

using the dankamongmen/fauxmemstream branch, ncwidth now runs properly when run without any arguments. both direct mode and rendered mode exit at interrogate_terminfo(), since there is no longer any backing terminfo (see #2009), and print a diagnostic to that effect. I need add a hardcoded set of Windows Terminal control sequence data, which is #2014, and at that point i think things will pretty much work on windows.

all the "run binary; it prints nothing and returns 1" was due to setupterm() failing, and the default being NCLOGLEVEL_SILENT. i've changed the default loglevel to NCLOGLEVEL_PANIC, so this kind of critical diagnostic is now visible unless explicitly locked out. i've excised the setupterm() call (and indeed all dependencies on ncurses) from the windows build. we're just about ready to go, assuming we can get our linking problems resolved on the bigger binaries.

[dankamongmen]

[dankamongmen]

i think we'll likely be running on windows by wednesday.

[dankamongmen]

we are now something on windows