Improved semantics for RtL text

[eyalroz]

The FAQ says:

Notcurses doesn't honor the BiDi state machine, and in fact forces left-to-right with BiDi codes.

Why? I mean, sure, BiDi takes some effort, but why categorically declare non-support? As opposed to "this is a low-priority fix" or such language?

[dankamongmen]

Why? I mean, sure, BiDi takes some effort, but why categorically declare non-support? As opposed to "this is a low-priority fix" or such language?

It's a great question! I'm absolutely open to revisiting this, but let me reference the thinking on it thus far:

896

1204

850

and i'll comment a bit further below.

[dankamongmen]

Essentially, it comes down to the following fact: Notcurses is a multiplanar model, and shifting text directions seem very difficult to integrate with multiple planes.

Let's say we have the characters 0, 1, 2, ...9 (note i mean here index 0..9, not the arabic digits 0 through 9).

LTR: 0123456789 RTL: 9876543210

all well and good. no problems here. perhaps we've written these to a 10x2 plane, so we have:

/----------\
|0123456789|  plane P0, 10x2, offset 0,0
|9876543210|
\----------/

now we add a new plane. this is 4x2, and it has two copies of the LTR row ABCD

/----------\
|ABCD|  plane P1, 4x2, offset 2,0
|ABCD| 
\----------/

so you now have this chunk of ABCD siitting atop the P0, yielding

98ABCD3210

so in this case, ought we reverse ABCD? probably not, right? but then is 98....3210 correct? remember, the sequence of input was 0123456789; it just appears as 9876543210 because it's RTL. so it seems just as valid for ABCD to have knocked out 2345 rather than 7654. but in that case, it should surely be 9876ABCD10, right? except that's not where we want ABCD.

add to this the fact that some terminals do RTL, and some don't, and it's impossible to know whether they will (indeed, in at least some it's a configuration option). so even if we try emitting 0123456789, hoping it'll be displayed as 9876543210, there's no guarantee of that.

sooooooooooooooooooooooooo the only thing that made sense to me was: never allow the terminal to do RTL. we enforce this by inserting hard LTR markers after each EGC with a bidi classification. this way, we always know that 0123456789 provided to the terminal will show up as 0123456789...and that 9876543210 provided to the terminal will always show up as 9876543210.

so, to answer your question, we do it because we don't otherwise know what the terminal is going to do.

NOW, surely we don't want to just shit all over our Arabic, Hebrew, Rohingya, Maldivian and shudder Mongolian friends, right? Well, they can just emit their characters in the order they want them to show up, assuming left-to-right motion. i.e. if you have  ތާނަ, emit it as  ނަ ތާ (no space), and it will show up exactly how you want it.

does that make sense? do you have a better suggestion? i only know a scant bit of Hebrew, so i'm not particularly well-informed, and would love good advice.

[dankamongmen]

oh, and give my love to חֵיפָה and the boys (and girls) down in Unit 8200! 🇮🇱

[dankamongmen]

oh hey shit, i read your "faster across the pcie bus" paper from back in 2017. i thought i recognized your name from somewhere!

[j4james]

My suggestion: Follow the same design pattern as the original RTL terminals, which is to provide APIs via which application developers can indicate their intent. Basically a way to say I'm writing RTL now, so move the cursor left after each character is output instead of right.

But you should never reorder characters that are already on screen, because you can't possibly know what they represent. Are two Hebrew characters next to each part of the same word, or are they from two separate windows with unrelated content that just happened to have lined up? Maybe they're just part of some kind of "ASCII art"?

The only person that knows what the characters represent is the app dev, so they should be the one that gets to decide what order they are output. Second guessing them just makes it impossible to do anything meaningful with RTL text. But you can make things easier for them, as long as you leave them in control.

Just my 2c.

[dankamongmen]

The only person that knows what the characters represent is the app dev, so they should be the one that gets to decide what order they are output. Second guessing them just makes it impossible to do anything meaningful with RTL text. But you can make things easier for them, as long as you leave them in control.

i agree with this wholeheartedly, and am doing this. without the LTR forcings, we don't know which way it'll be drawn. thus it is impossible for me to reliably fulfill the app dev's goals. by forcing LTR, i allow them to lay things out exactly as they want.

[dankamongmen]

now what i could add is something like ncplane_putstr_rtl(). but then we're Cartesianing our output functions by another dimension and that is gonna explode very soon.

[dankamongmen]

what gets ignored here a lot is BtT, as mongolian is written. here again, we allow it to be accomplished via exact positioning and a universal direction rule.

[j4james]

by forcing LTR, i allow them to lay things out exactly as they want.

Sorry, I should have said, I think what you're doing now is perfect. There's nothing stopping someone writing a bidi-aware application on top of a framework that stays out of the way. My suggestion was only if you wanted additional RTL-specific functionality on top of what you already have, and I didn't want you taking the auto-RTL approach that some terminals seem to think was a good idea.

[dankamongmen]

@eyalroz , what are your thoughts, having read this? perhaps the real change needed is one to the FAQ?

[eyalroz]

My thoughts are that I need to get to know the library more to better understand planes and multiplanarity - because I didn't get the intuition you intended to evoke with your example.

My intuition comes from real life :-P If I wrote ABCD on a small sign and put it on top of a bigger sign where I wrote 9876543210, at offset 2 from the left, then what I'll see from above is 98ABCD3210.

So it seems just as valid for ABCD to have knocked out 2345 rather than 7654.

why is that?

(You don't have to answer, I'll go read the other issue pages.)

[dankamongmen]

My intuition comes from real life :-P If I wrote ABCD on a small sign and put it on top of a bigger sign where I wrote 9876543210, at offset 2 from the left, then what I'll see from above is 98ABCD3210.

So it seems just as valid for ABCD to have knocked out 2345 rather than 7654. why is that?

because it depends on whether the plane was intended to cover cells 2, 3, 4 and 5 of the visual area or glyphs 2, 3, 4 and 5 of the text. for instance, take a look at the text in the first minutes of the Notcurses III hype video. if you look closely at the expository text, you'll see that it's being dimmed towards the back. new text is brightest, oldest text is dimmest.

i implemented this by placing a NCALPHA_BLEND plane atop the text plane, where this blending plane was dimmed according to the desired scheme. there, the upper left is darker than the upper right. if i had RtL text, i would presumably want that inverted, no? since the oldest text would be on the upper right. so here we want to make the decision based on the text layout. we want the darkest on the right, not on the left. this implies that i need to know the direction the text is flowing. see earlier comments: by relying on the terminal's arbitrary BiDi implementation, i cannot know this. thus the app dev must explicitly drive this layout, which is absolutely possible to do; you just write in reverse, or start from the right.

which is why i think maybe the wording in the FAQ is what's wrong. it's not that you can't write right-to-left as intended; it's that you must consciously do so, since otherwise we can't know whether your intention is being honored or not. if you write Gimel, Bet, Alef, it's not going to reorder it as Alef Bet Gimel, but if you write Alef Bet Gimel, it's also not going to be reordered as Gimel Bet Alef.

[dankamongmen]

here's another issue: let's say i have two 1x1 planes, one at {1, 0} and one at {1, 0}. the first has an Alef. the second has a Bet. I then move the second to {1, 1}. ought this show up as Bet Alef, or Alef Bet? if we honored RtL straight (and the terminal also did), a 2x1 plane into which one wrote "Alef Bet" would show Bet Alef. what of the logical 2x1 equivalent made up of 2x 1x1s? and what will the terminal do?

the only sane answer seems to be forcing LtR presentation, and allowing the app dev to place things precisely.

[eyalroz]

Well, once you have BiDi text, you need to be able to specify things like "start side" and "end side" rather than just "right" or "left". So, in the ABCD example - it depends on how you defined things. If you wanted "offset from left", you would say that and get that. If you wanted "offset from start side", you would say that. This is not the same as the app explicitly driving the layout: The app does not have to remember what the bottom plane text direction choice is, it can just say "align my second plane to the start side".

About the Alef-Bet planes - if they are two planes, then they don't constitute a single BiDi run. Aligning planes is not the same as uniting planes. It's just the same with, say, HTML table rendering. Even if there are no borders, cells are rendered individually. That being said - you could very easily create tricky examples; see the shocking Text rendering hates you if you want to be brought to tears...

[dankamongmen]

About the Alef-Bet planes - if they are two planes, then they don't constitute a single BiDi run. Aligning planes is not the same as uniting planes. It's just the same with, say, HTML table rendering. Even if there are no borders, cells are rendered individually. That being said - you could very easily create tricky examples; see the shocking Text rendering hates you if you want to be brought to tears...

i agree with this, but the terminal might not. test if you don't believe me: open a terminal, generate an Alef, hit enter, cup to the cell following the Alef, and enter a Bet. numerous terminals will swap the two glyphs (and others will not).

html would be easier to work with, but alas, that's not what i'm working with.

all such schemes break down because different terminals do different things. again, i bring up Konsole: it has a config option, which can be changed at runtime, as to whether it "honors" BiDi. i have absolutely no way of knowing what it's going to do at a given time if i have two RtL glyphs next to one another. the only way i can control things is by inserting LtR forcings to break up any possible RtL chain. i see no way around this, nor any way that user hints could help--the user must take desired order explicitly into account, and make up for it.

now, maybe there's a middle ground: when given a chunk of RtL text, i could lay it out myself RtL, just as i'm suggesting the user ought. that way, they needn't think about it. i've disliked this idea because it means

ncplane_putstr(n, "rtlfun");

and

for(const char* s = "rtlfun" ; *s ; ++s){ ncplane_putchar(n, *s); }

generate different output (the former would be written "nufltr", the latter "rtlfun"). so i put the question to you, oh speaker of the sinistral: would this be helpful? would it eliminate most of your annoyance? or would this end up being a false balm, one that suggests rich RtL support but doesn't actually imply it? ought i handle Mongolian text bottom-to-top?

[dankamongmen]

now, maybe there's a middle ground: when given a chunk of RtL text, i could lay it out myself RtL, just as i'm suggesting the user ought. that way, they needn't think about it. i've disliked this idea because it means

another thing that makes this unattractive, or at least worrying, is that i have no rich knowledge of right-to-left languages, and would be kinda shooting blind despite now being responsible for deciding what ought be RtL and what oughtn't (like, numbers are always LtR no matter what language they're surrounded by, right?). but those would of course just be bugs that can be reported and fixed.

if you think this scheme is a good direction to take, let me know, and i can code it up, probably for 3.0.

[j4james]

deciding what ought be RtL and what oughtn't

As far as I remember, the Unicode standard documents all of this stuff in great detail. But there are also libraries you can probably use that will do a lot of the work for you (I've not tried it myself, but I've heard good things about https://github.com/fribidi/fribidi).

Edit: This is the Unicode doc I was thinking of: http://www.unicode.org/reports/tr9/

[dankamongmen]

I've removed all calls to egc_rtl(), and the function itself. So far as I can tell, it wasn't doing its job anyway, and seems responsible for some problems in the whiteout demo when run in Kitty. So this whole argument is reset. With that said, it'd probably be wise to reexamine the matter in conjunction with a domain expert or at least a member of the Tribe.

As of 2.4.9, RTL is allowed to flow unmolested. For now.