Unicode problems on NixOS (glibc 2.26 - 2.27 locales incompatibility)

[maralorn]

I am not sure, if this is a bug or just a problem with my system. I cannot find a way to get notifications with non-ascii symbols to work.

My system is configured to use utf-8 everywhere and has no encoding problems with anything else.

This is a part of my env:

LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8

I am using

eventc 0.24.1 (using libeventc 0.24.1)

and a command like

eventc critical battery -d "title='Batterie lädt!'"

will fail with the message

(eventc:5279): eventc-WARNING **: 23:40:20.682: Couldn't parse the arguments: Invalid byte sequence in conversion input

Also bubbles not triggered from eventc but the dbus-notify plugin wont get shown, when they contain unicode.

I use nixos and have compiled eventd myself. Might it be, that I could some compile parameters wrong?

Any help on this would be much appreciated!

[sardemff7]

I am really surprised that such an error could happen. I actually don’t test eventd on non-UTF-8 systems so I would expect bug reports from them, if anything…

I tested your eventc command and it works here, so I would suspect an issue on your side, though the lack of error from any other software is surprising.

What is the output of localedef --list-archive on your system? If en_US.utf8 is not listed, you can try localedef -i en_US -f UTF-8 en_US.UTF-8, but I think it’ll be listed already.

Does locale output what you expect?

And last test, could you compile this test program and gives me the output:

/* gcc -o test-argv test-argv.c $(pkg-config --cflags --libs glib-2.0) */
#include <locale.h>
#include <glib.h>

int
main(int argc, char *argv[])
{
    setlocale(LC_ALL, "");
    for ( gint i = 1 ; i < argc ; ++i )
    {
        gchar *b = argv[i];
        gchar *e = b + strlen(b);
        const gchar *ve;
        g_print("argv[%d]: %s\n", i, b);
        if ( ! g_utf8_validate(b, e-b, &ve) )
        {
            gchar *v = g_utf8_make_valid(b, e-b);
            g_print("    Invalid UTF-8: %.*s ~= %s\n", (gint)(ve-b), b, v);
            g_free(v);
        }
        g_print("  ");
        gchar *p = b;
        for ( gchar *w = argv[i] ; w < e ; ++w )
        {
            gchar *c = b;
            if ( w > b )
                c = g_utf8_find_next_char(p, e);

            //g_debug("c %p == w %p", c, w);
            if ( c == w )
            {
                gchar uc[6] = {};
                g_unichar_to_utf8(g_utf8_get_char(c), uc);
                g_print("  (%s)", uc);
                p = w;
            }
            g_print(" %hhx", *w);
        }
        g_print("\n");
    }
}

[maralorn]

First of all: Thank you for your help!

I am kind of lost, which is probaby mainly caused by my missing knowledge of nixos. localedef --list-archive does not return. Don't ask me why … locale looks right:

/tmp/utftest> locale
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Your small program outputs:

[nix-shell:/tmp/utftest]$ ./test-argv abcäöüß
argv[1]: abc????
    (a) 61  (b) 62  (c) 63  (?) c3 a4  (?) c3 b6  (?) c3 bc  (?) c3 9f

[sardemff7]

I installed a NixOS VM to test myself, and I cannot reproduce the error. I can reproduce the localedef --list-archive endless loop though, but you cannot localedef -i en_US -f UTF-8 en_US.UTF-8 due to Nix design. And I see in the configuration file that they take care of that.

If you change my test program to that:

    /* replacing the setlocale(LC_ALL, ""); */
    char *r;
    r = setlocale(LC_ALL, "");
    if ( r == NULL )
        g_warning("Could not set locale");

I guess it outputs the warning?

At this point I’m pretty sure it is a system issue, and NixOS people would probably be more helpful. :-)

[maralorn]

Yeah, it outputs the warning. But also per default my LC_ALL is unset.

So you compiled the snippet under nixos and that worked fine with unicode?

[sardemff7]

Yes, it worked perfectly. LC_ALL in setlocale() is not the environment variable, it’s just a “set it all” value. It’ll loop over all the LC_* cases, check (in this order) $LC_ALL, $LC_<whatever> and $LANG (and probably fallback to C).

[maralorn]

Okay, I found it. (Thanks to a hint by @flokli) glibc-locales has an incompatibility between <= 2.26 and == 2.27 (see here).

Which is a problem under nixos if your environment is from the stable branch (nixos-18.03) with pre-2.27 glibc, but you build your applications from unstable with glibc 2.27 (which I had to do because of stable meson being to old) as noted here.

Just for reference: I put the following fix from here in my system config:

environment.sessionVariables = {
    LOCALE_ARCHIVE_2_27 = "${unstable.glibcLocales}/lib/locale/locale-archive";
  };

I appreciate very much the effort you took in helping me on this. I probably couldn't have done it without your input. Thanks!

[sardemff7]

No problem, nobody like bugs (except speedrunner maybe). And thank you for running the additional tests. I’ll update the issue title to ease future searches if anyone happens to have a similar issue.