Fix pango and vty frontends so they don't convert to String then back to ByteString

[acertain]

Currently, the pango/gtk+ frontend converts our Rope type (of utf8-encoded ByteStrings) into a String for the pango haskell library, then the haskell library converts it back into a utf8-encoded ByteString. This is a serious performance issue, especially without incremental rendering, and profiling puts this conversion as the most expensive operation.

There's an issue on gtk2hs's bug tracker here from 6 years ago.

[ethercrow]

Situation is similar in vty frontend, Rope is converted to String, although recent vty versions expose Text-based API. It is not straightforward to fix this, because frontend code has to annotate characters with colors. Current representation of annotated text is [(Char, Annotation)]. Maybe we need an efficient annotated Rope type.

[Fuuzetsu]

Can you show some profiling data? I have tried to profile Yi in the past but I am not experienced in the area so it was difficult for me to pinpoint actual problems.

[ttuegel]

The rainbow package has (what looks like) an efficient annotated text type. Maybe that approach would be useful here?

[Fuuzetsu]

I don't see it saying anything about being efficient but it's probably better than [(Char, Annotation)].

[ttuegel]

Efficient was my evaluation, not the original package author's. Performance should be similar to lazy Text, which is essentially a tail-lazy list of strict Text chunks anyway.

[Fuuzetsu]

The rope package has Data.Rope.Annotated, perhaps we could use that. It's unclear to me at first glance how to use it but I think it should do.

[acertain]

I can't seem to reproduce my profiling result.

Doesn't look like rope will help here much, I think the right way is probably a IntervalMap (also for Yi.Buffer), but gtk+ and vty don't support them (afaik).

[Fuuzetsu]

I have a branch where I implemented my own rope over Text. There is already a speed and memory improvement but it is not quite read to land yet, there are a ton of places where we used Text. It should be more or less ready (if not optimal) in next couple of days. The reason I didn't end up going with rope is that it's not finished and a lot of the parts are inefficient. There are a lot of conversions it does because it uses ByteString internally and close to any action we would want to perform requires us to convert. In contrast, my solution just uses (large) chunks of Text which handles encoding natively and is known to be Pretty Damn Fast. See yi-rope repository. It has benchmarks in case you want to try to optimise it further which I encourage as it will cause flat speedups throughout Yi.

We still don't have a solution to [(Point, Char)]. I think we can however reduce annotations separately from the string in the buffer and splice it in at the last moment. This is in contrast of what we do now: take the whole string and do an ‘in place’ replacement. I only considered this today however and I can not try until all the other rope changes I have land.

[Fuuzetsu]

Pango no longer deals with either ByteString nor String. We instead use YiString all the way until we send it off to the pango library at which point we convert to Text which is not very expensive and in common case we drastically clip the content anyway (see recent commits for details). Taking off the pango label.