Will `cat /dev/urandom` corrupt my display?

[dpc]

Hi, one thing that was always bothering me about unix terminals is that cat /dev/urandom and similiar could corrupt the display.

It seems to me that "untyped stream of bytes" was never the right abstraction for communication between processes and interactive terminals.

I have not give it too much thought yet, and I'm looking for opinions and resources, but it seems to me that if every stream of bytes was typed (with some default), like eg. prefixed with mime type (defaulting to binary data stream) or simper-http-like protocol, then both communication between processes and between process and terminal could be much robust.

Eg. legacy process like cat, streaming bunch of bytes would output bunch of binary adta and notty would know that all data sent contains no escape codes, displaying eg. hex dump of it, and responding with no "type header" on cat's stdin.

New-world interactive processes, when outputting anything would start with a \0notty/v1\0 header, or something like that, to which terminal would respond with \0notty/v1\0 on stdin, confirming that process is attached to tthe terminal.

The scheme could be completely different.

[JanLikar]

I partially agree, but how often do you have a need to deal with random binary data printed to the terminal? I don't think it is very often.

[cheery]

"Untyped streams of bytes" is all we got right now. It's part of how our filesystem works. It has turned out to be simple, although not optimal. Time has shown we can live with that and trying to force changes into that would result in a shitfest now.

Notty remains a text terminal while extending it with new escape codes. This is rather backwards-compatible way to work. Although I'm not sure how programs are aware that they can use the interface introduced by notty.

The proposed features would be partially redundant, as there is isatty that already tells whether there's terminal associated to stream.

Notty is much like a form of TeX output. It presents what you write to it. I think I like this part of it. It's much better than trying to introduce type annotations into streams and try to pretend that it's typed stream.

In current setting this feature doesn't come from notty. It comes from the implementation of cat you're using, that already knows whether it's interfacing with a terminal or not.

[dpc]

I would argue that filesystems are quite much better as binary blobs carry: name (with extensions, which is kind of a type), metadata, xattrs, permissions etc.

Example of isatty is exactly what I don't like about unix terminals: it's a magic out-of-band syscall interface, which makes stuff like telnet, all the pty magic etc. necessary. With in-band typed protocol, one could use raw tcp connection as telnet, or an encrypted one as ssh. etc.

Anyway, I'm not trying to change development of natty - it's just natty is one of very few project that is actually addressing this level of our infrastructure, so I'm sharing my thoughts and looking for opinions.

I'm aware it seems much impractical, but I'm tempted to purse a prototype of the model I'm describing (with some other changes I'm thinking about). I guess I'd just have to write a simple custom shell, and custom terminal (that would be the biggest item, and maybe I could fork of natty, since a lot of things are already working here very well, and I'd "just" have to change the protocol), and adapter for existing software, a library for new one and some examples of new tools.

[withoutboats]

Hey, sorry I missed this discussion until now.

In general, I agree with this sentiment:

It seems to me that "untyped stream of bytes" was never the right abstraction for communication between processes and interactive terminals ... it seems to me that if every stream of bytes was typed

The issue, of course, is how to achieve this.

First, I want notty to be a well-behaved ANSI terminal when paired with a program that anticipates that its writing to an ANSI terminal. I think this is mandatory behavior for a new terminal to gain adoption. So cat /dev/urandom will indeed junk your terminal, because to behave otherwise would not be consistent with the behavior of cat. I can't think of a solution to this particular pathological behavior that doesn't break anticipated positive behavior; surely we don't want cat to hexdump all data, what about when I cat README.txt?

Basically, I think the solution to this lies on the other side of the tty. notty is developed with the goal of being able to be told in relatively high level terms how to display some data (e.g. "this is an image"; "this is JSON"; "this is hexdump"; "this text has this semantics [highlight it appropriately]") and then do the right thing. There is some other program the user will run which will build a different sort of shell environment, probably by imposing a protocol on top of UNIX primitives that already exist; when this environment needs to write to the terminal, it will be able to use notty's escape codes to format the data properly.

As to how programs know that they're talking to notty, you still need to use isatty to know you're talking to a terminal (I don't like it, but its what we have), but all terminals set the TERM environment variable, and programs go off of that, similar to the User-Agent request header in browsers. Programs are also expected to use this to look up a terminfo sheet, though nowadays this system is largely cruft and nearly every terminal has the same terminfo sheet (notty's is in the repo, it is the same as xterm-256colors minus 1 or 2 features notty doesn't support).

I haven't looked into how well terminfo supports arbitrary new entries, I think the thing we'll want to do when notty is eventually made more stable is to specify a notty version in the terminfo sheet, and the running process is responsible for knowing how to implement that version of notty. terminfo's comand description language isn't expressive enough to describe all of notty's commands.

[dpc]

I agree. It's definitely not only a local issue that notty could solve on it's own. I'll close the issue for now, but if anyone has any comments, I'll be following the discussion, and I'm still thinking about this.