x/term: create subrepo for terminal, console, pty and tty related packages

[davecheney]

Following the lead of #13058 towards smaller, more focused, sub repositories, I would like to propose the creation of a sub repo for packages related to handling the interaction with tty like things. This includes

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.
• windows cmd console

Much of the code for this exists in various packages, some in other sub repos like x/crypto/ssh/terminal, others has been duplicated incompletely by package authors with a specific use case; most times without a comprehensive story for windows.

Various informal proposal have been made, like #12853, relating to posix termios.

Other proposals like #12101 for the creation of an x/sys/terminal group of packages have been rejected on the grounds that sys is too low level, which also dovetails into the desire for more smaller sub repos, rather than larger ones.

There is also the the long standing question of why the code to disable terminal echo lives in the x/crypto/ssh package.

With the exception of the name of the repository, which I'm sure will require debate, is there general support for a sub repository along the lines outlined in this proposal ?

[mattn]

:+1: x 256

[sbinet]

yes, please. especially the pty feature (which currently lives under github.com/kr/pty) which is always a bit subtle to handle right.

[rsc]

I am not sure about this one: terminal handling may not be significant enough for its own subrepo.

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...). It's certainly true that the ssh terminal code does not belong in crypto.

That said, I don't know where else to put it either. Let's see if any other suggestions arrive.

[gdamore]

yes! I am willing to help here. I'd like some higher level primitives, and frankly expressing things in terms of POSIX termio functions would be really nice. It could even be possible to express Windows console APIs in such terms.

In addition to just enabling / disabling raw & cooked mode (and echo/noecho), I see specific needs around:

• baud rate determination (used for real serial lines, but even console programs need it -- for example curses type apps need to know "baud" rate when calculating padding character delays required in terminfo strings
• tcflush & tcdrain
• serial port manipulation (stop bits, parity, etc.)

I'd far rather this all be expressed in terms of POSIX termios than some other way, at least to applications. (Using ioctls or whatever under the hood is fine.)

[bradfitz]

@rsc wrote:

In general the subrepos seem to be mirroring our top-level standard library directories (sync, net, crypto, ...).

Perhaps in that spirit we create golang.org/x/os then, for operating-system specific packages. I have a bunch of code myself in various repositories I'd love to centralize, for things that differ between Linux/Mac/Windows. Terminals seem like it could go under x/os.

[distributed]

Yes, please. I'm the author of github.com/distributed/sers, a way too noncomprehensive serial port library. I basically wrote it because I needed to set "non standard" baud rates to speak with hardware.

I would be glad to help out, should the proposed subrepo become reality.

[kim-racktop]

I have some preliminary code I wrote for the Posix tc functions that I have tested on Illumos. I could contribute that if people think it would be useful. I created a branch of x/sys in my local copy and have the code in x/sys/term.

[peterh]

With respect, I disagree that there isn't enough terminal handling to warrant its own subrepo. The termcap/terminfo parser alone would probably be a decent size, in addition to all the other things already mentioned. Plus all the os-specific versions of the interfaces for output (colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-, etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner) none appear to support terminfo (or... is there a Plan 9 equivalent?). That's probably because everybody independently figured it wasn't worth the effort to write a terminfo parser for a single project. If the library and the effort are going to be shared, I'd be more likely to help.

[gdamore]

If we go beyond just termios, into terminfo parsing and so forth, yes, this would be quite a bit bigger.

I wrote a fairly complete terminfo parser in tcell see github.com/gdamore/tcell

(Its a little special purpose, because I don’t pick up all terminfo keywords for storage, because most terminfo capabilities are useless in modern terminals. But I have a full parser and expander, including a complete implementation of TParm and TGoto.)

The low level half of tcell (basically everything in the base tcell package) is probably what you’d want if you’re going to be doing rich terminal work. This API may still be a little higher level than we’d like to express in the core Go library though. (For example it has fairly rich color mapping support, mapping any RGB value into the supported colors of the terminal, including best-match color on 256 color terminals.) The entirety of the base tcell package is about 7KLOC, plus a few KLOC for the preparsed embedded terminfo database. (Basically I precompile the most likely terminals into the code in an attempt to minimize dependencies on external database files.)

On Fri, Oct 30, 2015 at 11:52 AM, Peter Harris notifications@github.com wrote:

With respect, I disagree that there isn't enough terminal handling to warrant its own subrepo. The termcap/terminfo parser alone would probably be a decent size, in addition to all the other things already mentioned. Plus all the os-specific versions of the interfaces for output (colours/styles/cursor positioning) and input (Ctrl-, Alt-, Fn-, Arrow-, etc keys) on non-Unix-like OSes.

I looked at the links in the proposal, and (like my own liner https://github.com/peterh/liner) none appear to support terminfo (or... is there a Plan 9 equivalent?). That's probably because everybody independently figured it wasn't worth the effort to write a terminfo parser for a single project. If the library and the effort are going to be shared, I'd be more likely to help.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-152615292.

[frederich]

I quite agree like Brad x/os.

[gdamore]

One problem I have with x/os, is that if we think of this functionality as “os-specific”, then it becomes more likely that there is drift between the interfaces.

That is, I agree that the implementation is OS-specific. But having OS-specific APIs is highly undesirable, and is devastating for application portability. (Indeed, this problem is exactly why things like POSIX and the standard C APIs exist …)

So while the location is not terribly important to me — especially if it is just base functionality like termio handling, if the location means that its likely that the APIs will not be compatible across different ports, I consider that a failure.

On Fri, Oct 30, 2015 at 12:50 PM, Jens Frederich notifications@github.com wrote:

I quite agree like Brad x/os.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-152635342.

[rthornton128]

+1. I also agree that functionality needs to be as OS agnostic/unified as possible akin to the majority of the standard library.

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

[davecheney]

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

[gdamore]

Indeed. I could use that functionality and I believe the work I'm doing in tcell today goes substantially towards addressing the other higher level needs. (With views, widgets and so forth.)

Sent from my iPhone

On Oct 30, 2015, at 10:35 PM, Dave Cheney notifications@github.com wrote:

Would a package like this also include higher level constructs akin to curses panel, menu and form or would that be a community extension with the standard library package being solely low level?

I would vote no to that. I think we'll be struggling with the scope as it is.

However, I would hope this package would provide all the raw materials needed to construct such a library.

— Reply to this email directly or view it on GitHub.

[techtonik]

It would be also nice if the package exposed performance metrics // tests both for terminal and for windows-like console.

[davecheney]

@adg @rsc ping

The x/time and x/sync repos have been approved, what are your thoughts on adding an as yet to be finalised, repository for terminal/serial/tty type packages ?

[adg]

I think x/os is probably the way forward.

[davecheney]

@adg golang.org/x/os sounds good to me !

[kim-racktop]

As long as we strive for an API that is consistent across the different OS's, sounds ok to me.

[gdamore]

Yes please!

That would NOT be consistent with the other stuff in x/os however. Mostly x/os seems to be a kind of dumping ground for OS-specific functionality, rather than any effort to provide a unified portable interface. For this reason, I’m much more in favor of an x/term (or probably better, x/tty, which would help keep the scope from ballooning into terminfo, curses, or anything like that).

On Wed, Nov 4, 2015 at 11:47 AM, Kim Shrier notifications@github.com wrote:

As long as we strive for an API that is consistent across the different OS's, sounds ok to me.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-153843047.

[adg]

That would NOT be consistent with the other stuff in x/os however. Mostly x/os seems to be a kind of dumping ground for OS-specific functionality, rather than any effort to provide a unified portable interface.

Some unix-isms crept in for historical reasons, but on the whole the os package is in fact a platform-agnostic interface to the operating system.

[techtonik]

Also, x/term and x/tty are already unixisms.

[davecheney]

I agree, but this that is unavoidable.

I'm sure there will also be packages that only make sense for the Windows command subsystem.

I hope that in addition to these operating system specific packages we can develop high level ones

For example, console control providing cursor control for text mode interfaces, Garrett has shown that this is possible for Windows with the same high level api.

Similarly I want to see a high level serial Comms package that operates across Linux, Mac and Windows. It's exported api should be agnostic, but will probably call on operating system specific packages to provide the implementation. See my GitHub.com/pkg/term package.

On 5 Nov 2015, at 19:07, anatoly techtonik notifications@github.com wrote:

Also, x/term and x/tty are already unixisms.

— Reply to this email directly or view it on GitHub.

[techtonik]

@davecheney Windows console is more powerful/responsive in terms of user experience, because it provides a separate channel for controlling console window object (independent orthogonal API from C standard streams), while in terminal it is all garbled into one stream, which is slow and inefficient (and requires complex state machine to maintain). So decide carefully based on data if you need this Comms port and why - maybe there is a better way..

[gdamore]

The Windows console abstraction isn't that much more powerful. And yes a higher level API can be created to handle all that. I know as I've done it for Tcell.

It would still be useful to emulate some low level termios functions though. Even Windows has the notion of raw and cooked mode for example.

Sent from my iPhone

On Nov 5, 2015, at 5:47 AM, anatoly techtonik notifications@github.com wrote:

@davecheney Windows console is more powerful/responsive in terms of user experience, because it provides a separate channel for controlling console window object (independent orthogonal API to C standard streams), while in terminal it is all garbled into one stream, which is slow and inefficient (and requires complex state machine to maintain). So decide carefully based on data if you need this Comms port and why - maybe there is a better way..

— Reply to this email directly or view it on GitHub.

[techtonik]

@gdamore I'd prefer to have user level API that can be later used to design proper low level interface in newer subsystems that require deterministic user interfaces (wayland etc.). Meaning that some "cooked" concepts need to be forgotten to avoid the same traps of the past.

Tcell - https://github.com/gdamore/tcell - looks cool. But it needs some quick intro / tutorial how to try it on your system.

[gdamore]

I don’t see how we can eliminate fully the idea of “cooked” vs “raw” terminals. That is, most of the times people expect terminals to respond to readline() (or whatever) by echoing character presses, and buffering until a newline is pressed. But when richer applications, or even prompting for a password, you need something more basic. You can do passwords by just disabling echo, but for full UIs with menus & so forth, you generally want to turn off buffering altogether.

On Thu, Nov 5, 2015 at 2:51 PM, anatoly techtonik notifications@github.com wrote:

@gdamore https://github.com/gdamore I'd prefer to have user level API that can be later used to design proper low level interface in newer subsystems that require deterministic user interfaces (wayland etc.). Meaning that some "cooked" concepts need to be forgotten to avoid the same traps of the past.

Tcell - https://github.com/gdamore/tcell - looks cool. But it needs some quick intro / tutorial how to try it on your system.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-154221023.

[davecheney]

@adg @rsc now that we have https://go-review.googlesource.com/#/c/16551/ in addition to the proposal outlined above what is your opinion on createding a golang.org/x/os sub repo ?

[adg]

As I said, I am in favor of x/os. @rsc is the decider.

[alexbrainman]

How are we going to explain difference between golang.org/x/sys and golang.org/x/os to users? What is the difference?

Alex

[davecheney]

How are we going to explain difference between golang.org/x/sys and golang.org/x/os to users? What is the difference?

I'm not really sure. This proposal, or a previous one suggested added terminal/tty/console style packages to the x/sys repo but this was rejected by @rsc. This proposal is a replacement, so I guess we can use the rational from https://github.com/golang/go/issues/12853#issuecomment-152556261

[rsc]

I fear x/os will become a kitchen sink. What's the line there?

[davecheney]

@rsc I cannot disagree there. Would you reconsider a proposal for x/term as originally detailed in the proposal ?

[techtonik]

x/os is cross-platform (meaning "Go's operating system interface for humans") and x/sys is system-specific?

[davecheney]

I like that definition. We've seen several reviews that suggest that x/sys not platform agnostic

On Tue, 10 Nov 2015, 19:44 anatoly techtonik notifications@github.com wrote:

x/os is cross-platform (meaning "Go's operating system interface for humans") and x/sys is system-specific?

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-155359467.

[techtonik]

If rephrased as common Operating System interface for humans then if something is not included in this concept means that no common interface between operating systems is possible.

To maximize the impact, the next step would be to create a pathway for explanations. So that people can get the answer "why it is not possible to get common interface", which will lead to description of actual conflicts, which is a necessary step to rethink things, which is a source data for improving human experience with Operating Systems (http://www.worldusabilityday.org/ is in 2 days), which could explain what could be changed, which is a necessary step to design new generation of Operating Systems and a source data to implement new hardware, which can also be based on completely different principles (optics, neural networks, etc.). But somebody need to start gather a technical database of compatibility conflicts, so that every new language out there didn't waste time on the same cross-platform problems that are not related to the language itself.

[rsc]

I don't believe x/os is what you really want here. I could see putting tty handling code into there, because the OS is involved with that, but not anything involving escape codes, so x/crypto/ssh/terminal would still not belong. The goal was to find a home for that code, right?

[davecheney]

@rsc

The goal was to find a home for that code, right?

Yes, the goal is as specified in the original post at the top of this issue

I would like to propose the creation of a sub repo for packages related to handling the interaction with tty like things. This includes

• posix ttys and ptys
• serial termios control, like terminals and rs232 devices.
• windows cmd console

[bradfitz]

At dotGo, @jfrazelle promoted on stage the acronym TUI (pronounced "too-ey") for Text User Interface, like GUI (Graphical User Interface). x/tui would encompass @davecheney's list of items, without having "term" in the name. :-)

[davecheney]

There are heaps of names that aren't really suitable, termios, tty, console, serial, etc. Of all, I preferred the x/os suggestion.

fwiw. I think x/time isn't a great name either, as it contains things concerned with rate limiting ... the obvious conclusion would be x/ratelimit. What's the point of this digression ? The name probably isn't critical.

[bradfitz]

FWIW, I find this name discussion very frustrating as well. I'd like to be producing and reviewing and sharing code instead of naming things.

[davecheney]

Thanks Brad. I'm pretty confident that any name chosen would be wrong by some measure. I think x/term would have been fine, but x/os is even better.

I think this issue shows a genuine need, terminal handling, echo disabling, serial port handing code has been rewritten, incompletely, dozens and dozens of times, on an as needs basis by package authors. It would be unfortunate if the go project couldn't provide a solution to this very common set of problems because we couldn't agree on the perfect name for the repo those packages lived in.

Thanks

Dave

On Wed, 11 Nov 2015, 08:23 Brad Fitzpatrick notifications@github.com wrote:

FWIW, I find this name discussion very frustrating as well. I'd like to be producing and reviewing and sharing code instead of naming things.

— Reply to this email directly or view it on GitHub https://github.com/golang/go/issues/13104#issuecomment-155571263.

[kim-racktop]

Well, naming things is one of the two hard problems in computer science. The other one being cache coherency and off by one errors.

That said, I am aware of this sort of discussion going on for at least 3 months. I am hoping that some name can be settled on reasonably soon and that the README.md file for the new repo describes why the name was chosen, as well as what goes into the repo and why.

I feel under-qualified to push for any particular suggestion. While I don't think the perfect name has been put forward yet, I don't have any other good suggestions and the ones already mentioned aren't too offensive.

[rthornton128]

Naming has the feeling of long term consequences and misconstrued intent, which is the crux of this discussion. Despite feeling not feeling constructive there is value in it nonetheless.

I feel that tui is misleading in that it invokes a higher level of abstraction than this package intends to offer. "Term" is common to Unix whereas "console" is common to Windows.

I have to agree with Dave in that x/os makes the most sense but does feel like a dumping ground. Perhaps, like std/os, this is the right place for os specific terminal code and leave an os agnostic implementation for a separate, higher level tui package?

[rsc]

re @bradfitz

FWIW, I find this name discussion very frustrating as well. I'd like to be producing and reviewing and sharing code instead of naming things.

We spend time getting names right not because we love arguing about names but because the name of something can have a very large impact on how easy it is to understand and how it will evolve.

In this case, the name has significant impact on what people will think belongs in the subrepo. It should be clear that x/term and x/os would be very different subrepos. I'm sorry about the holdup here, but of course producing and reviewing and sharing code can happen in other places too, if it's more important to move fast and break things than to get things right.

[gdamore]

I really think tui is terrible for the reasons you list and I'm pretty unhappy about x/os because it does feel like it becomes a dumping ground.

Let me offer up x/tty

While nobody actually uses teletypes anymore this interface used with stty and the termios ioctls on UNIX systems covers all the bits one needs for handling tty devices while remaining low level.

Furthermore it will be possible to emulate this functionality easily for Windows. The idea of setting raw mode, disabling or enabling buffering and echo are pretty universal. Windows consoles can offer some enhanced functionality as well but for something like the needs of ssh that is entirely unnecessary. (Extras like double buffering and mouse reporting.)

I cannot think of any system which offers a prompt like interface for shell interpreters which cannot supply an implementation of these interfaces.

I'd be equally happy if this were called termios and the interfaces built to emulate POSIX termios. I actually believe that for most consumers that will be the easiest and most natural API.

Btw ssh does need escape sequences or anything like that. Just low level termios support is sufficient.

Sent from my iPhone

On Nov 10, 2015, at 5:02 PM, Rob Thornton notifications@github.com wrote:

Naming has the feeling of long term consequences and misconstrued intent, which is the crux of this discussion. Despite feeling not feeling constructive there is value in it nonetheless.

I feel that tui is misleading in that it invokes a higher level of abstraction than this package intends to offer. "Term" is common to Unix whereas "console" is common to Windows.

I have to agree with Dave in that x/os makes the most sense but does feel like a dumping ground. Perhaps, like std/os, this is the right place for os specific terminal code and leave an os agnostic implementation for a separate, higher level tui package?

— Reply to this email directly or view it on GitHub.

[davecheney]

I'm fine with x/tty.

[adg]

What about x/console with package 'console' being in the repo root?

[gdamore]

I like x/console less, but can live with it. console suggests (to anyone other than a Windows console user) a specific interface -- which can overlap with but is also orthogonal to tty handling. (On UNIX the "console" is the primary login interface by default, and is also where kernel messages and high priority syslogs are usually directed; it has a specific interface /dev/console that is typically not interactive but just a sink for messages.)

To be honest, any of /x/tty, /x/console, /x/term, /x/termio[s] can work. I'm partial to /x/tty and /x/termio because I think those most accurately reflect the functionality we plan to put there, without suggesting that a lot of other quasi-related stuff (like ANSI escape sequences) wind up there.

[adg]

Would ANSI escape sequences not be in this repository? Maybe I misunderstand the scope.​