submit kernel patch to consume APCs and Privacy Messages

[dankamongmen]

See #2050, especially @klamonte's comment at the bottom. We ought go ahead and do our civic duty and submit a patch to properly consume these in the kernel, based on the Williams automaton. This isn't a Notcurses bug per se, but it will make our lives easier.

[dankamongmen]

DCS, also

[WSLUser]

Can you be sure to link the mail thread once you submit a patch? I'd be curious (along with I'm sure every terminal and TUI developer) in the discussion with the kernel folks.

[dankamongmen]

from vt100.net:

ANSI Control Strings When the terminal receives any of the escape sequences listed below, it responds as usual to control characters received (octal 000 -- 037 and 177) and discards any printable characters received (octal 040 -- 176). The terminal reverts to text processing mode when one of the following conditions occur.

• ESC \ (octal 033 134) is received,
• CAN (cancel), SUB (substitute), or ESC (escape) is received,
• an error occurs.

Name	Mnemonic	Sequence
Application program command	APC	ESC _	033 137
Operating system command	OSC	ESC ]	033 135
Privacy message	PM	ESC ^	033 137
Device control	DCS	ESC P	033 120

[ghost]

If S8C1T is enabled (via DECSCL (CSI {62 to 65} ; 2 p), or ESC {space} G), then APC, PM, OSC, and DCS can also be ended with 0x9C (8-bit ST).

Also if S8C1T is on, 0x90, 0x98, 0x9D, 0x9E, and 0x9F can start DCS, SOS, OSC, PM, and APC respectively. SOS (ESC X or 0x98) is like PM and APC.

Reference: http://ftp.invisible-island.net/xterm/ctlseqs/ctlseqs.html

[dankamongmen]

i'm working on the patch now. i'm only going to make the initial one as complex as i can slip through without a great deal of explanation, as i really don't want anyone to balk and start talking about adding unit tests etc. so the first iteration might just cover the stuff i have above, and maybe even just APC.

[dankamongmen]

alright, i've got a very simple, very minimal patch which passes my basic tests (those being, "if i revert the change to avoid sending the kitty query on linux, do we no longer get bleedthrough of an APC?" and "if i send an APC and end it with an ST, followed by crap, does the crap show up?"). i'm gonna go ahead and mail this off and see what happens.

[dankamongmen]

sent, waiting for it to show up on LKML archives....

[dankamongmen]

Aug 18 23:49:11 vps postfix/smtp[1566224]: 4A4FC2B98A: to=<torvalds@linux-foundation.org>, relay=ASPMX.L.GOOGLE.COM[172.217.215.26]:25, delay=0.36, delays=0.01/0.02/0.07/0.25, dsn=2.0.0, status=sent (250 2.0.0 OK  1629344951 e60si1287601ybi.468 - gsmtp)
Aug 18 23:49:11 vps postfix/smtp[1566226]: 4A4FC2B98A: to=<gregkh@linuxfoundation.org>, relay=ASPMX.L.GOOGLE.COM[172.217.215.26]:25, delay=0.36, delays=0.01/0.05/0.13/0.16, dsn=2.0.0, status=sent (250 2.0.0 OK  1629344951 g6si628218ybu.383 - gsmtp)
Aug 18 23:49:12 vps postfix/smtp[1566227]: 4A4FC2B98A: host vger.kernel.org[2620:137:e000::1:18] said: 450 4.2.0 <linux-kernel@vger.kernel.org>: Recipient address rejected: Greylisted for 60 seconds (in reply to RCPT TO command)
Aug 18 23:49:12 vps postfix/smtp[1566225]: 4A4FC2B98A: to=<nickblack@linux.com>, relay=mxb.mailgun.org[44.240.110.196]:25, delay=1.4, delays=0.01/0.03/0.29/1, dsn=2.0.0, status=sent (250 Great success)
Aug 18 23:49:13 vps postfix/smtp[1566227]: 4A4FC2B98A: to=<linux-kernel@vger.kernel.org>, relay=vger.kernel.org[23.128.96.18]:25, delay=1.7, delays=0.01/0.06/1.6/0.06, dsn=5.4.3, status=bounced (host vger.kernel.org[23.128.96.18] said: 553 5.4.3 Hello [173.230.130.29], for MAIL FROM address <dank@schwarzgerat.orthanc> the policy analysis reports DNS error with your source domain. (in reply to MAIL FROM command))
Aug 18 23:49:13 vps postfix/cleanup[1566223]: 0F0CD2B9FE: message-id=<20210819034913.0F0CD2B9FE@vps.qemfd.net>
Aug 18 23:49:13 vps postfix/bounce[1566228]: 4A4FC2B98A: sender non-delivery notification: 0F0CD2B9FE
Aug 18 23:49:13 vps postfix/qmgr[1541636]: 0F0CD2B9FE: from=<>, size=7385, nrcpt=1 (queue active)
Aug 18 23:49:13 vps postfix/qmgr[1541636]: 4A4FC2B98A: removed

bah why do i run my own mail server

[dankamongmen]

bounced it, ought work now

[dankamongmen]

https://lkml.org/lkml/2021/8/19/4

[dankamongmen]

once 5.14.0 drops, i'll resend. i doubt it's getting picked up when we're already at -rc7. when i do so, i'm also going to hit a few of the people who've more "recently" (as in since 2010) touched it.

[WSLUser]

Surprised nobody even said anything yet. Then again, I've seen some patches take months before being even looked at. Reaching out to the folks who last touched the vt code probably is good way to get ball going for it. I'd like to grab the complex version of your patch if you've got it ready "in the wings" once this simple version is merged so I can use it the next time I compile a kernel for WSL2 (I'm already applying custom patches from MS and Clear Linux, so adding better VT support for my custom kernel would hardly be an issue).

[dankamongmen]

Surprised nobody even said anything yet.

it was late in the release cycle, a bad time to send it. this code doesn't get touched very often looking at the git logs, and someone checking it would need to go consult documentation (it's not "obviously right"), so i'm not very surprised. Linus just dropped 5.14, so i resent it (after fixing up some checkpatch.pl warnings, which were another reason why it was probably ignored, erp). i just sent it again. probably ought have dropped a v2, RESEND in there but oh well.

https://lkml.org/lkml/2021/8/30/148

[j4james]

someone checking it would need to go consult documentation (it's not "obviously right")

Technically it's not right, because the VT10x devices didn't support any of those control strings, and the kernel VT driver is essentially a VT102 emulator. 😉 I doubt anyone cares about nit-picking like that though.

If anything, you might get push back regarding the overhead of those three extra states, which I'm assuming you don't really need if you're just planning to ignore them all? I think they are quite fanatic about performance in the kernel.

The other thing that would concern me would be backwards compatibility. With the billions of systems out there that are using Linux, you can be sure that there's a script somewhere that's unknowingly outputting one of those introducers, and this patch is going to suddenly "hang" that device until another ESC or BEL comes along. We've had a few complaints along those lines in Windows Terminal, although I think all of those have been regarding the C1 controls, which at least wouldn't be a problem for your patch.

[dankamongmen]

If anything, you might get push back regarding the overhead of those three extra states, which I'm assuming you don't really need if you're just planning to ignore them all? I think they are quite fanatic about performance in the kernel.

while i have not inspected it myself, i'd happily bet you 50 dollars or european dollars or pounds sterling or florins or warm loaves or Victory Cigarettes that any optimizing compiler written in the past 20 years optimizes those down to the same thing

[dankamongmen]

The other thing that would concern me would be backwards compatibility. With the billions of systems out there that are using Linux, you can be sure that there's a script somewhere that's unknowingly outputting one of those introducers, and this patch is going to suddenly "hang" that device until another ESC or BEL comes along.

shhhhhhhhhhhhhh it's a secret to everybody

[j4james]

while i have not inspected it myself, i'd happily bet you 50 dollars or european dollars or pounds sterling or florins or warm loaves or Victory Cigarettes that any optimizing compiler written in the past 20 years optimizes those down to the same thing

OK that statement was guaranteed to make me a go and check, but you were right. Some of the things I thought might be issues weren't a problem, and in retrospect that should have been obvious. But you might have lost on a technicality, because there is still additional code overhead to set the three separate states - with one state, the jump table for the switch statement just points all three characters to the same place. I can't see that having any significant affect on performance though.

[dankamongmen]

But you might have lost on a technicality, because there is still additional code overhead to set the three separate states - with one state, the jump table for the switch statement just points all three characters to the same place. I can't see that having any significant affect on performance though.

even that would surprise me. any kind of value analysis atop controlflow+dataflow analysis of the SSA style ought be able to see that all three values reach the same control flows, and the value is never otherwise sampled, and fold them down into a single path. i would be disappointed otherwise!

[dankamongmen]

even that would surprise me. any kind of value analysis atop controlflow+dataflow analysis of the SSA style ought be able to see that all three values reach the same control flows, and the value is never otherwise sampled, and fold them down into a single path. i would be disappointed otherwise!

(my background is primarily compilers and hpc, fwiw =])

[j4james]

even that would surprise me. any kind of value analysis atop controlflow+dataflow analysis of the SSA style ought be able to see that all three values reach the same control flows, and the value is never otherwise sampled, and fold them down into a single path. i would be disappointed otherwise!

I'd be seriously impressed if the compiler could do that, but then again I have a long history of being seriously impressed by compiler optimisations, so I'll take your word for it. I was only looking at the assembly produced for the stand alone function - I'm assuming this would require link-time optimisation.

[dankamongmen]

...and, it's merged. from the Chief Penguin himself:

I've applied the patch as-is since I'm starting my merge window work            
with "random patches in my queue".                                              

I'm not 100% sure we care all that much about trying to make the                
console be more vt100-compatible, considering how long we've been at            
the current state.                                                                                                                                              
But I have to admit to being slightly emotionally attached to us doing          reasonably well on this side. One of the original things Linux did was          
console emulation, even before it was an OS.. 

[dankamongmen]

https://github.com/torvalds/linux/commit/3a2b2eb55681158d3e3ef464fbf47574cf0c517c