[bisected] Locale broken with nixpkgs on Fedora {29…31}

[phi-gamma]

Issue description

Affected package: mutt. The affected system is nixpkgs on Fedora 29 (Linux 5.0.9-200.fc29.x86_64).

The problem manifests itself by Mutt not displaying properly encoded UTF-8 text.

Bisection identifies this commit as the culprit:

$ git bisect good
d966f31f23b5cb37c0e6bd8553f8503c0465f205 is the first bad commit
commit d966f31f23b5cb37c0e6bd8553f8503c0465f205
Author: Jörg Thalheim <joerg@thalheim.io>
Date:   Tue Jan 22 22:08:44 2019 +0000

    glibc: add support for C.utf-8

:040000 040000 99a9210dc0a37d097ca21f065780a715ba23c82a cf56ff55d0098e818517560fd85a51ab311187a5 Mpkgs

Mutt up to that point handles UTF-8 just fine.

Steps to reproduce

Install mutt via nix-env and run it in xterm under en_US.utf8 locale.

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

- system: `"x86_64-linux"`
- host os: `Linux 5.0.5-200.fc29.x86_64, Fedora, 29 (Twenty Nine)`
- multi-user?: `no`
- sandbox: `no`
- version: `nix-env (Nix) 2.1.3`
- channels(philipp): `"nixpkgs-18.09.1594.1a4498d1328"`
- nixpkgs: `/home/philipp/.nix-defexpr/channels/nixpkgs`

[FRidh]

cc @Mic92

[Mic92]

What does $LOCALE_ARCHIVE points to in your case?

$ echo $LOCALE_ARCHIVE
/run/current-system/sw/lib/locale/locale-archive

[parisni]

I guess I am having the same concern with both mutt/neomutt on both ubuntu/archlinux: accents are displayed as ???

[phi-gamma]

Fwiw the problem persists in 19.09. Mutt is practically unusable.

What does $LOCALE_ARCHIVE points to in your case?

It is empty / undefined.

As mentioned above, this happens using nixpkgs on Fedora where there is no /run/current-system. There exists a system locale-archive but

$ LOCALE_ARCHIVE=/usr/lib/locale/locale-archive mutt

does not fix the issue nor does

$ LOCALE_ARCHIVE=/nix/store/0lyi1vqhccvglrn2byvk3qa46isc4vd0-glibc-locales-2.27/lib/locale/locale-archive

with any of the glibc related paths in the nix store.

[Mic92]

I don't have a neomutt/mutt setup. Do you have a command/configuration that I could run to show the encoding problems on a non-NixOS machine?

[parisni]

I suggest you have to setup a mailboxe, and put a test email in it with special character. maybe this setup can be shared as an archive

[phi-gamma]

Do you have a command/configuration that I could run to show the encoding problems on a non-NixOS machine?

It should show up with mutt running a minimal configuration: https://paste.centos.org/view/224cda7a

Save this file as muttrc and fill in the blanks (server, login, pw). Install mutt with nix-env, then invoke it as mutt -F muttrc. In case you’re not dropped into the inbox folder, type the c key, then ? to obtain a list. Move up/down (arrow or vim keys should work) and select the inbox with spacebar.

Once in the inbox the issue will be obvious from message subjects that contain non-ascii codepoints.

[phi-gamma]

Fwiw I just noticed that weechat suffers from the same issue if built from nixpkgs on FC31. Same package is fine on nixos. Likewise, the dedicated FC31 package (which I can’t use directly because FC31 doesn’t ship the Matrix plugin) runs fine on FC31.

[phi-gamma]

Investigating this some more I noticed that in a nix-shell, $ LC_ALL=C weechat reproduces the issue; LC_ALL=C.utf8 weechat fixes it. The system locale is en_US.utf8; outside the shell, running locale(1) gives no error. Inside the shell it gets interesting:

[nix-shell:~]$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
[… snip a bunch of LC’s]
LC_ALL=

I don’t have any of the LC_* variables set in my environment and the nix-shell is pure anyways.

Why does the C.utf8 locale work while the en_US.utf8 doesn’t?

[Mic92]

In a nix-shell it works fine, if glibcLocales is added as well

$ nix-shell --pure -p bash --run 'LC_ALL=en_US.utf-8 locale'
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=
LC_CTYPE="en_US.utf-8"
LC_NUMERIC="en_US.utf-8"
LC_TIME="en_US.utf-8"
LC_COLLATE="en_US.utf-8"
LC_MONETARY="en_US.utf-8"
LC_MESSAGES="en_US.utf-8"
LC_PAPER="en_US.utf-8"
LC_NAME="en_US.utf-8"
LC_ADDRESS="en_US.utf-8"
LC_TELEPHONE="en_US.utf-8"
LC_MEASUREMENT="en_US.utf-8"
LC_IDENTIFICATION="en_US.utf-8"
LC_ALL=en_US.utf-8

nix-shell --pure -p bash -p glibcLocales --run 'LC_ALL=en_US.utf-8 locale'
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=
LC_CTYPE="en_US.utf-8"
LC_NUMERIC="en_US.utf-8"
LC_TIME="en_US.utf-8"
LC_COLLATE="en_US.utf-8"
LC_MONETARY="en_US.utf-8"
LC_MESSAGES="en_US.utf-8"
LC_PAPER="en_US.utf-8"
LC_NAME="en_US.utf-8"
LC_ADDRESS="en_US.utf-8"
LC_TELEPHONE="en_US.utf-8"
LC_MEASUREMENT="en_US.utf-8"
LC_IDENTIFICATION="en_US.utf-8"
LC_ALL=en_US.utf-8

As this adds LOCALE_ARCHIVE:

nix-shell --pure -p bash -p glibcLocales --run 'echo $LOCALE_ARCHIVE'
/nix/store/da0h55nhsipysryz9zcb9i7chbqffa45-glibc-locales-2.30/lib/locale/locale-archive

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[phi-gamma]

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

Adding glibcLocales to the shell indeed fixes the issue. Though this raises the question as to whether that dependency needs to be explicitly added to any package that depends on glibc and does some kind of text processing. Locales are a non-optional part of the C standard and while it’s great to be able to drop the heavyweight dependency where you know it’s irrelevant, it should not be absent in the default context.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[Geometer1729]

This fixed a similar issue for me https://nixos.wiki/wiki/Locales