Parse arbitrary ANSI escape sequences

[thomasballinger]

Someone in #bpython wanted to be able use their pretty colored text representations of objects in bpython. When I tried it in bpython, that particular serious of escape sequences didn't work - curtsies couldn't parse it. The curtsies escape sequence parser should be made more reliable so it can handle escape sequence-laden strings it didn't create.

[thomasballinger]

Arbitrary ANSI escape sequences should be able to be parsed into FmtStr objects.

[jquast]

It's pretty insane, but I generate regular expressions for the given TERM of all known terminal capabilities, separated into groups of will and won't move, for the purpose of measuring a string's length.

https://github.com/jquast/blessed/blob/master/blessed/sequences.py#L72-231

I also add generic sgr's to that list, https://github.com/jquast/blessed/blob/master/blessed/sequences.py#L280-290

This allows to discover the width of strings blessed did not generate, https://github.com/jquast/blessed/blob/master/blessed/tests/wall.ans

[thomasballinger]

I just saw that, I've got a blog post I want to publish about how blessed is generally awesome :) Such a list would be useful here, but I'd want to drop any sequences that can't be represented as part of a FmtStr - if it's not foreground color, background color or style drop it.

[philer]

Is there any progress on this? It's been quite a while. Apparently it's been blocked for almost 2 years by https://github.com/thomasballinger/curtsies/issues/57 (curtsies).

Here's what print("\033[33;3m orange and italic \033[0m") does for me:

1. On the latest version from pypi (bpython 1.6.0 on curtsies 2.11) it dies with an exception
2. Installing curtsies directly from github (as suggested in #624) gives me version 2.10 (??). This version doesn't die but apparently strips all unknown escape sequences completely.

I'm not quite sure why you parse those out (something about string lengths?). I'd expect those escape sequences to make it through untouched. After all there aren't just colors but also CSI codes that do actual cursor control. The only means of cursor control that remains is "\r". That's barely enough to do a progress bar (see tqdm).

Is there a way to have bpython/curtsies leave escape sequences unmolested? I love using bpython as a repl but this makes it pretty much useless when it comes to testing rich command line applications.

Either way, keep up the great work!

[thomasballinger]

@philer, the reason these are parsed out is that bpython does its own formatting, and more importantly, its own layout in the terminal. So line wrapping, for instance, is done explicitly by bpython: asking the terminal how wide it is, then doing the math. So yeah, string length is a big part of this. Terminal resizing is where this is particularly important, especially combined with undo: we need to know where to erase to etc. Essentially bpython (the curtsies version anyway - try python -m bpython.cli for something a little less dramatic about stripping formatting) declaratively renders each rendered frame, like a virtual dom library.

I should definitely check out the curtsies version weirdness, definitely not good that it's dying with latest pypi versions, but can't promise a time to look at it. I did recently get cleared to work on bpython on work equipment, so hopefully my GitHub contributions section will green again up soon :)

At this point getting around this by really changing up how rendering works would be considerable work. So the workaround of parsing the sequences ourselves into the data structure that bpython/curtsies use to respresent formatting text is less work. This would make bpython work better for a "reasonable subset" of csi codes, which probably would not include cursor control. Cursor control gets interesting and complicated when you're trying to provide a nice repl experience, because all of a sudden the cursor is supposed to have started somewhere else, so your prompt should be somewhere else, rendering previous outputs gets harder, etc.

Your best bet for testing rich command line applications is probably the vanilla python interpreter, because the new IPython interace probably has similar issues, though they might not be as bad (I'm guessing here, you might try it!). I'm a big fan of rich command line interfaces, and have felt this pain as well. But unfortunately dealing with arbitrary command line sequences is out of scope for bpython, the most we'll get is colored/styled text.