make menuconfig not supported on Windows

[mbolivar]

Since the kconfiglib switch, make/ninja menuconfig can't work on Windows, but this page implies it's possible run menuconfig on Windows:

http://docs.zephyrproject.org/application/application.html#overriding-default-configuration

A fix for this issue likely involves writing a Python3 GUI on top of kconfiglib that allows users to graphically edit their configuration.

[carlescufi]

Options for the GUI library to use:

• https://github.com/peterbrittain/asciimatics
• https://github.com/jonathanslenders/python-prompt-toolkit

[mbolivar]

A text user interface would indeed be a replacement for menuconfig, no disagreement there.

IMO a "real" GUI would be very nice to have as well. I normally use xconfig instead of menuconfig, as its UI is a lot better (example: I can search for an option, then enable it in one click from the results window).

[carlescufi]

@mbolivar you would be able to do that using asciimatics or python-prompt-toolkit as well I'm pretty sure. The advantage with text mode is much smaller dependencies and consistent layout

[ulfalizer]

Hello,

@mbolivar A jump-to feature like that is something I've been missing in menuconfig as well.

It should be pretty easy to implement in Kconfiglib, by looking up the symbol by name in kconf.syms and jumping to one of its menu nodes in kconf.syms["FOO"].nodes (a symbol can get multiple nodes from being defined in multiple locations, though you usually only see one).

If you wanted to get fancier, you could iterate through kconf.defined_syms instead and search prompt texts and help texts and things like that as well.

@carlescufi Those look pretty cool. At least ASCIIMATICS and prompt_toolkit seem to have mouse support, which is another thing I've been missing in menuconfig.

You've probably already seen it, but there's a menuconfig example/proof-of-concept at https://github.com/ulfalizer/Kconfiglib/blob/master/examples/menuconfig.py. What's missing for a basic usable menuconfig is menu tree navigation (right now it just dumps the entire tree) and less clunky input.

Ask if you have any questions.

[mbolivar]

@ulfalizer awesome, thanks for the pointers! I wish I had more time to work on this :\

[carlescufi]

Here's a "wish list" for this particular feature:

• Python3-based tool that includes all of menuconfig's functionality
• Uses one of the ncurses replacement libraries above
• Has full mouse support
• Tested on *NIX and Windows 7/10

[mbolivar]

Options for the GUI library to use:

https://github.com/peterbrittain/asciimatics https://github.com/erikrose/blessings https://github.com/jonathanslenders/python-prompt-toolkit

I've done some initial review of these based on their documentation and installing them and playing around with their basic examples.

In all three cases, the libraries appear to be thoughtfully designed and carefully implemented.

Blessings provides basic terminal coloring and text styling, as well as positioning and fullscreen control. The feature set seems too restricted to implement the full wish-list, which includes features like text boxes and input fields, mouse support, etc., without a lot of trouble.

Asciimatics, by contrast, provides TUI support, including buttons, line drawings, and (terminal permitting) mouse input, out of the box.

Prompt toolkit has all of the above, as well as support for advanced features like text autocomplete, Emacs and Vi keybindings within the TUI, and more.

I would therefore rule out blessings and restrict the search to Asciimatics and Prompt toolkit.

Between the two, prompt toolkit seems to have a much better ecosystem, with many significant projects using it as a backend. I'm a longtime user of the IPython shell, for example, and I was interested to learn it uses prompt_toolkit. Some of the other features people have implemented on it (like the aws shell) were futuristic by CLI standards -- really impressive stuff.

Asciimatics is Apache 2.0, prompt toolkit is BSD 3 clause.

Given that, I think prompt toolkit is a clear winner, as something that can be used to start small but eventually produce really killer features.

[mbolivar]

Given that, I think prompt toolkit is a clear winner, as something that can be used to start small but eventually produce really killer features.

Ugh, though there's some pain coming down the pipe:

https://github.com/jonathanslenders/python-prompt-toolkit/issues/568

[carlescufi]

Ugh, though there's some pain coming down the pipe

Not sure this is a real problem, it looks like he'll push to pypi without changing the name, and even if he did change the name we could use the *2 version?

[ulfalizer]

Nice coincidence. I just bumped Kconfiglib's major version to 3 (though I doubt anyone will be affected by the small API behavior change -- just sticking to principles since I said I use semantic versioning). :P

Yeah, seems best to go with the *2 version. Thanks for the investigation!

[mbolivar]

Not sure this is a real problem, it looks like he'll push to pypi without changing the name, and even if he did change the name we could use the *2 version?

Lots of tools depend on the 1 version, I'm not sure how distro package management would cope with both installed in one system -- I guess 'import prompt_toolkit' would work with the version 1 from apt in your IPython install, but the *2 version in your pip --user install in the Zephyr environment?

More seriously, testing things in the early documentation pages beyond hello-world from the 2.0 branch broke fairly often. It seems like the docs are in some kind of intermediate state between the two versions, which could be a pain to develop against.

[ulfalizer]

Probably wouldn't be a huge pain to switch out prompt_toolkit out for something else at least, if it turns out to be a pain. Could do a basic usable version with just keyboard input and simple menu drawing first, and then see what to go with from there when adding more fancy features like mouse support. The basic version would probably be pretty easy to port over to other toolkits if needed.

[ulfalizer]

Would be cool if there was a UI toolkit that could generate both a text-based UI and a graphical UI as output, depending on settings. Doubt there's anything like that though.

[ulfalizer]

Here's some notes from playing around a bit with some of the toolkits (I might update them as I go):

ASCIIMATICS

The API and widget set seem nice and flexible, and mouse support is good.

Unfortunately, there's really noticeable startup delay and (worse) input delay in the UI examples I tried (samples/contactlist.py, samples/forms.py, samples/treeview.py).

Holding down a key in the contactlist.py example, I saw CPU usage go up to 90% on my Core i7-2600k. I think ASCIIMATICS might have been more of an animation library initially, with the UI stuff added later.

Here's two related issues I found:

• ENH: samples for common UI forms seems slow (no background animation in the examples I tried though, afaics)

• Startup performance seems pretty bad

Python Prompt Toolkit

Seems to be focused on text input, with some widgets tacked on.

Mouse support might only be available for text widgets and simple buttons:

• Clicking checkboxes and radio buttons in examples/full-screen/yes-no-dialog.py doesn't do anything. Clicking the <YES>/<NO> buttons at the bottom works though, and the mouse also works for the menu at the top.

• The mouse doesn't work at all in example/full-screen/buttons.py

There's also the upcoming 2.0 version switch that @mbolivar mentioned, which seems a bit messy.

Urwid

Haven't looked into it much, but it seems it might lack Windows support.

The standard curses module

Pretty low-level and do-it-yourself.

Would be snappy at least, and probably has the most complete mouse support. Drawing some simple Windows manually might not be that much work (I'm kinda leaning towards a "fullscreen" display, with just some buttons at the bottom and a title bar at the top, and maybe a dialog or two).

Unfortunately it doesn't work out of the box on Windows, though getting it ro run might be possible. Some related links:

• https://github.com/general-ai-challenge/Round1/issues/17 (seems to use https://www.lfd.uci.edu/~gohlke/pythonlibs/#curses)

• https://github.com/jmcb/python-pdcurses

Some other links:

• https://stackoverflow.com/questions/32417379/what-is-needed-for-curses-in-python-3-4-on-windows7/34438835

• Unicurses (might have an issue with 64-bit Windows)

Blessings

A thin wrapper around the standard curses module, with the same limitations. Also seems to use some ioctl() stuff for terminal resizing that might not work on Windows.

[ulfalizer]

Made a little progress on a version that uses the curses module. It's still in a WIP (hacky) state and missing essential features (no save/load, symbol information, search, mouse support, etc.)

The curses-specific parts are trivial, so it'd be easy to switch out curses for something else. I suspect that will be true when more features are added as well. Most of the complication is in menu navigation logic and the like.

Some things I like about curses:

• Zero dependencies on *nix (the curses module is in the Python standard library).

• Fast and snappy (pretty much instantaneous response). Unresponsive/flickering UIs is a personal pet peeve.

• Super flexible. A bunch of other libraries use it as a backend.

It would be nice if PDCurses turns out to be a viable option on Windows. Then we could use curses everywhere.

I added some Vi keybindings by the way, in addition to the "intuitive" keybindings:

• J/K: Down/up

• L: Toggle item or jump into menu

• H: Jump to parent menu

Tell me if you have any other feature requests. :)

[carlescufi]

@mbolivar @SebastianBoe @nashif @ulfalizer

I have looked through the option of using PDCurses and the .whl file you point to, but here are the issues with it:

• We don't have the source code for it for the latest Python versions
• It's unsupported, placed in some random person's homepage
• If a problem arises on a user's computer (say a segfault) we are on our own and we would need to provide our own wheel
• It doesn't support Unicode
• It is awkward to install (download the wheel for the particular Python version and architecture)
• It might disappear in the future or not be supported for upcoming Python versions, leaving us hanging

Therefore I would much rather use a pure Python alternative that gives us the certainty that we will be able to support it for many years into the future.

If there are limitations with the Python toolkit that we choose to use it's OK, we can improve that in the future slowly but surely. I'd much rather not have all the features (like mouse support) out of the box than take a dangerous bet of using an unsupported API on Windows that depends on a precompiled binary we don't have control of.

[carlescufi]

A few more comments on the options you list:

• Urwid does not work on Windows, requires several modules not available on it, so not an option
• Blessings doesn't seem to support color on Windows, comes with no examples either, it's a very small project
• Unicurses: Effectively a dead project

For me the best contender is still python-prompt-toolkit by far because of all the reasons listed so far.

[ioannisg]

@gbakke FYI

[ulfalizer]

Summing up some notes from IRC:

• FWIW, the guy providing the Python PDCurses wheel seems to be pretty serious. But yeah, there's of course the risk that he stops updating them.

• The user would need to install the right version of the wheel as well.

• I wonder what form the Unicode brokenness in the Python PDCurses wheel takes. Slightly messed-up display vs. very messed-up display vs. not being able to input Unicode stuff at all would make a difference I think.

• python-prompt-toolkit is about to release version 2.0, which isn't on PyPI yet. @carlescufi noted that it can be installed directly from the GitHub repo with e.g. pip install -e git+https://github.com/jonathanslenders/python-prompt-toolkit@2.0#egg=prompt_toolkit.

• All I really need is a text output buffer + input events. I'm going for a simple fullscreen interface that would be intuitive to people used to make menuconfig, and I think a fancy UI toolkit is overkill for that (and would get messy if the widgets don't fit the model). The interface is just a list, some buttons, and a few modal dialogs.

  I wonder if python-prompt-toolkit can be used like a portable curses.

All the work I'm doing now would apply for any toolkit (i'm basically only using getch() + addstr() on the curses side), so I could get a usable version done in curses first and then switch over to something else if needed.

[carlescufi]

@ulfalizer thanks for the updates. If all you need is getch() and addstr() then I am pretty sure we can get a version of those relatively easily either using PPT or taking the necessary code from it.

[ulfalizer]

@carlescufi I suspect it might be messy to rip it out cleanly if they don't provide it in the interface. There's probably display optimization and stuff going on. I think that's needed to avoid flicker even in terminal emulators.

Pretty sure we'll work something out at least. :)

[ulfalizer]

Added scrolling, support for menuconfig FOO symbols, and some misc. minor things:

Not winning any beauty contests yet.

One thing I've always found annoying about make menuconfig is that you can't search for a symbol and then jump directly to it, so I designed it to make that easy to implement:

The current position is remembered only as the currently selected node cur_node (plus a preferred_row variable, to keep the scroll from jumping around when you go in and out of menus). Everything else is derived from cur_node (e.g. the current menu). cur_node can then be set to wherever you want to jump to, even if it's in a completely different menu.

[carlescufi]

@ulfalizer looking good. A bit more coloring and this will be menuconfig. If you want I can give it a go on Windows, need the branch.

[ulfalizer]

@carlescufi It's in a bit of a half-broken state at the moment, but I'll send you a version to try out tomorrow at the latest. :)

[ulfalizer]

Got Windows installed and did some testing. Works fine until you resize the terminal.

Unfortunately, fixing that might be a bit painful. Some notes:

No KEY_RESIZE events are returned by getch() in the Python code, so the terminal resize isn't even detected. PDCurses itself supports KEY_RESIZE, but the wheel does not. Functions like curses.resizeterm() and curses.is_term_resized() end up not even existing.

It might be because setup.py from the wheel only has this:

define_macros = [("WINDOW_HAS_FLAGS", None)]`

Looking at _cursesmodules.c in the wheel source code (which is just a modified version of _cursesmodule.c from the CPython sources), it looks like HAVE_CURSES_IS_TERM_RESIZED and the like would need to be defined for those functions to become available. Maybe those macros are supposed to get defined by the extension building framework when building extensions though...

Perhaps rebuilding the wheel (python setup.py build) with some extra flags would fix it at least.

[ulfalizer]

I added a simple configuration dialog for string/int/hex values (no input verification yet) and save/load functionality, so it's in a basic usable state now.

Still ugly as sin, but that's for later.

Some missing/planned stuff:

• Symbol information

  I'm thinking of a format similar to the one generated by genrest.py.

• Search and "jump to"

  I want a list of matching symbols to pop up as you type. Selecting a symbol will jump directly to it in the menu tree.

• "Show all" mode

  This would show symbols even if they aren't currently visible. That would make the search nicer, as "show all" mode could be turned on automatically if the symbol you jump to isn't visible.

  It would probably be a nice feature to have in other cases too.

Please tell me if there's any other features you want to see.

[carlescufi]

@ulfalizer This is excellent progress. the "Search and jump to" functionality is something I've always wanted in menuconfig so this will be most welcome. Regarding the beautification, I would settle with a simple color scheme reasonably similar to what's in menuconfig today, but maybe @mbolivar, @nashif or @SebastianBoe have opinions on the visual style that we should adopt here. In any case, nice work.

[mbolivar]

maybe @mbolivar, @nashif or @SebastianBoe have opinions on the visual style that we should adopt here.

I wish I had opinions :).

I don't use menu-driven applications from the terminal much (I prefer xconfig to menuconfig).

My only general idea would be that discoverability, for users who are accustomed to full-featured GUIs would be preferable to magic key sequences to get help or otherwise start to figure out how to use the application.

[ulfalizer]

Hello,

Status update is mostly that I wasted time on nitpicking alternative scrolling implementations and trickiness related to that. Scrolling is surprisingly tricky to get robust, with having to remember positions when jumping in and out of menus, nodes disappearing, windows being resized, scroll offsets, page up/down, the window being resized while a dialog is open, etc.

I'll try to get back to more essential stuff soon so I can get a first version out. Should really have left some of this till later, but you know how it is when you double down on stuff.

@mbolivar Any particular features in xconfig that you've missed in menuconfig?

[ulfalizer]

Got the scrolling and resizing worked out nicely. Pretty simple codewise now too. Always get a false start or two.

Progress update below.

• Cleaned-up dialogs and colors

  Yellow really was the least ugly there, though I'm sure it can be tweaked.

  

• Input dialog that also shows valid ranges

  Don't think the standard menuconfig shows the range.

  Supports cursor movements (including horizontal scroll), Unicode, and "the usual" editing keys.

  

• Error checking

  Unexciting stuff that needs to be there.

  

  

  • Help display

    You can probably tell this needs some work interface-wise. ;)

    

  • Robust and seamless resizing

    Including with dialogs open. They "float" and move above the main display.

Missing features:

• Manual .config loading

  ...as opposed to the automatic loading of .config when you start the configuration interface.

  This gets a bit tricky since the currently selected symbol (or the entire menu) might "disappear", so I thought I'd add it after I add a search feature that can jump to a symbol, as it would also need to deal with symbols that aren't (currently) visible.

  The plan is to add a "show all" feature and turn it on automatically when you end up at a symbol that isn't visible.

  • Search with "jump to" (see above)

  • "Show all" mode (see above)

  • Mouse support

  • Whatever I've forgotten

I'm thinking of releasing an initial version without those. It's basically in a usable state now, with just the "10%" stuff left (plus whatever I forgot).

[carlescufi]

Hi Ulf, awesome update. I think it's time to post a DNM/RFC PR now so we can start discussing this. The PR should also include the following:

• Integration with CMake (at this point I suggest just using menuconfig.py instead of mconf with the same framework as the current one)
• Removal of C tools (we really want to get rid of those in one shot)

In a follow-up PR then we can target:

• Switch from source to gsource
• Simplification of kconfig.cmake

I think the for CMake side @SebastianBoe can probably contribute, so we are all in sync from the get go

[ulfalizer]

Status update:

• Refactored the color/style code to be easy to maintain. The first version was ad-hoc and spread out.

  • Fixed crashes and other issues when colors and other optional features aren't supported. I tested with TERM=vt100, since it's likely to work with most terminals if it works with that.

  • Added verification of ranges, so you get an error if you attempt to set an int/hex symbol to a value outside an active range.

  It isn't strictly illegal to set an out-of-bounds value (though the value will be ignored while the range is still in effect), but I decided that it's a good idea anyway.

  • Made visible if properly hide menus.

  • Did a little bit of work on the help display dialog.

  • Lots of nits all over.

Getting closer.

[ulfalizer]

Status update:

• Made an information interface in the same style as the main display.

• Added information texts for all the different types of items (symbols, choices, menus, comments). It should have most of the stuff from make menuconfig now, barring stuff I've missed.

• Tweaked expression printing to show the values of all the operands, like make menuconfig does.

  I also made it point out when a symbol is undefined, as make menuconfig displays that in a super-confusing way (S390[="S390"] means S390 is undefined there, while it now shows up as S390(=undefined/n) now).

Screenshots below:

As seen in the last screenshot, long expressions can get ugly, which is a problem make menuconfig has as well. The expression could probably be split it up in some way to fix it, but it might not be a priority at the moment.

Major TODOS off the top of my head:

• Check that everything still works on Windows.

• Clean up and let the dust settle a bit.

Tell me if you spot something you miss.

[ulfalizer]

There's now a do-not-merge pull request at https://github.com/zephyrproject-rtos/zephyr/pull/7140. Feedback appreciated!

[ulfalizer]

A proper pull request is now at https://github.com/zephyrproject-rtos/zephyr/pull/7174.

[carlescufi]

This has now landed in master

[mbolivar]

Nice work @ulfalizer!