Unicode input combining characters not working

[jbirnick]

When I type a ü in dmenu-rs, it doesn't show the letter but some weird symbol (making clear that it's an unsupported letter or so).

Note: I'm using the US-international keyboard layout. So to type a ü, I first type something that gives the two dots, and then a u, and this is automatically combined to a ü

[Shizcow]

Have you checked that the rendering font supports this character? Running echo "ü" | dmenu correctly renders the character on my system with default font settings.

[jbirnick]

Yeah the font is not the issue. The command echo "ü" | dmenu correctly renders ü for me as well. It's just about the input. The issue is to combine the two unicode characters into one I guess. I think it's this character ̈ (see here) or this ¨ or this ¨ which is followed by a u to produce ü. But the input is not working correctly and it produces this character � instead of a ü.

In summary: It's not a rendering issue. Instead, the input is not working correctly; the two unicode symbols combined do not yield ü but �.

[jbirnick]

You can try it out for yourself. If you have US layout, this should work:

• press Shift + Alt Gr (to the right of SPACE) + the key with ' and "
• (let go)
• press u

This should produce ü. If you do it in e.g. Firefox or Chrome, it should even show you just the two dots after the first step. But in dmenu it doesn't produce a ü.

[Shizcow]

My attempts to repro

On my system, I don't have bindings to produce the combining character but I can paste into dmenu and see some attempt at merging the symbols. Pasting in this character, then pressing LEFT (to focus on the text point that allows for combining diacritics) I get this:

If I then press u I get:

There's that space after the ü character, which cannot be removed (for some reason).

The above screenshots were with a non-default font. Using my system's default mono font I get:

Which no longer has that space, but the diacritic is placed incorrectly.

Using DejaVu Sans, I get:

Which looks perfect to me.

Questions for @jbirnick

Seeing this behavior on my system, I'm still not convinced this isn't a font issue. I'm pretty certain that all the font rendering (diacritic merging included) is X's job, and this program simply passes through the raw input strings straight to X for rendering. Even in the case of special/merged unicode characters, it's still up to X to get the rendering right. Theoretically this couldn't be an input issue, but let's still work together to get this figured out.

Let's do two sanity checks to see if your system behaves the same as mine:

• What font are you using?
• Can you still repro with echo "_ü_" | dmenu --font 'dejavu sans'?
  • The input character in that command is the diacritic-combined version. But you can also try in the input box.

Failing that: Could you show me a screenshot of what you get? I see you posted the � symbol, but I want to rule out that you meant something other than the question-mark-in-diamond symbol (and that you meant some other symbol but my Firefox is just deciding not to render your entry).

And if you still need help: Can you repro this issue with the original C++ version of dmenu? If not, that could help me fix this more quickly.

[jbirnick]

Okay, I guess I sort-of found it. You mentioned X11 and that reminded me that this is an X11 application... I use wayland and so I run it with xwayland.

So I tested in on X11. It didn't work (expected, because I assumed my X11 input settings are incorrect), but interestingly enough the behavior even was different than through xwayland (unexpected). (On X11 it shows me a u, with xwayland it shows me indeed the � question-mark-in-diamond symbol.

So I guess what I have to do is to adjust my input settings in X11, and then it might work.

But.. well, I'm too tired for that. Input management is one reason I'm over X11, I don't want this headache again. If you ever decide no write a wayland-native version that would be great! In any case thank you for the efforts you put into this issue.

PS: If you want to try out the real thing (not just pasting some characters), you can set your keyboard layout to the altgr-intl variant of the us layout, and then follow the key instructions from my previous post.

[jbirnick]

Nevermind, I still think there is an issue!

Because I just realized how a good part of my applications are running through xwayland, and they are doing just fine. For example, Chromium. So I think it's actually a dmenu-rs-specific input issue.

[Shizcow]

Ahh, wayland. Makes sense. This project is X-native, but I'm glad to hear it mostly works with wayland under compatibility settings.

There are currently no plans to make this project wayland native, and it's extremely unlikely I'll ever pick this up in the future. I've tried coming back to this project a few times, but it doesn't hold my interest enough to do such large development.

If you ever figure out how to get it running properly under Wayland let me know. It would make a good section in this project's README to help others out.

[Shizcow]

Because I just realized how a good part of my applications are running through xwayland, and they are doing just fine. For example, Chromium. So I think it's actually a dmenu-rs-specific input issue.

Test with the font settings (echo "_ü_" | dmenu --font 'dejavu sans') and let me know what it does. Chromium uses a unicode friendly font as default, where dmenu uses a very barebones monospace font by default.

[jbirnick]

Your command gives the ü just fine. I'm in fact using the DejaVu Sans Nerd font by default. You can trust me, it's an input issue and not a rendering issue.

Last info maybe, if you're interested: I now went back to X11 and tried it in other applications, like Chromium. It does not work there. This is a good sign though. My final stand is:

• my X11 input settings don't have the altgr-intl variant enabled, so I can't do my ü stuff. that is, in no application, when I'm running X11. if I would enable it (for which I'm too lazy), then I assume it will work fine on X11, well definitely in Chromium and stuff, and I assume in dmenu-rs as well, given you're saying that you don't even handle the input yourself
• now to wayland. when I run Chromium with xwayland, then interestingly enough the ü input works. it seems to somehow use the wayland input but render through X11 or so.
• when I use dmenu-rs through xwayland, then it seem to use neither the exact X11 input (because as I said it spits out a different character) nor the wayland input (because then ü would work)

So I guess you/we could now start a journey on how input is managed through xwayland... or perhaps we just leave it at those interesting observations. :)

[jbirnick]

Oh and for compatibility with wayland, I guess everything works except those two:

• when the focus goes away from dmenu-rs, it just "freezes" (i.e. stays open), but there is no way to get the focus back to the dmenu. this can be quite annoying, I always have to do killall dmenu.
• well the input issues I just described

[Shizcow]

it's an input issue

Agreed, I'm just trying to figure out if it's an input issue on the window manager's side or dmenu's side. It sounds like wayland is not sending the correct keystrokes to the application. This project uses XNextEvent to handle keypresses as individual characters. The specific part we're interested in is String::from_utf8_lossy(&buf[..len as usize]) -- this gets the input returned from XmbLookupString. I think that's just all of the yet-to-be-processed text sitting in X's input buffer.

We can rule out any rusty shenanigans (from_utf8_lossy) by checking what happens when you try the exact same thing using the original C++ dmenu. Can you build that from source and try again? If it does still repro, then we know it's an X11<->Wayland thing. If it doesn't repro, then it should be an easy fix; I'll ask you to attach some debug code, dump the X11 input buffer in both the Rust and C++ code, see what it looks like after processing, and tweak the Rust code to match.

Very interesting to hear the different behavior in Chromium et. al. handling this correctly. It's possible they use a different Xorg input function that's wayland compatible. dmenu-rs was written referencing the original dmenu code, the core components of which are extremely old. If this is the case, it'll be a simple fix.

While a wayland-native / wayland-native-compatible rewrite is out of the question, I can totally help out with getting the compatibility layer working correctly.

[Shizcow]

As for the focus problem -- I'm not surprised this program freezes. It never expects focus to go away. It uses X11 calls that grab the keyboard in a very low level way, meaning that it's supposed to be impossible for focus to be pulled away. The intended behavior of dmenu is to be as "always on top" as possible. Wayland messing with this is unsurprising. I've never done any Wayland development, so I'm not sure if (A) there's some Wayland call that can do this or if (B) it's even possible to detect wayland and issue wayland-specific calls under xwayland.