search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.25k stars 13.51k forks source link

Change the order of locale archive search path #85823

Open kirelagin opened 4 years ago

kirelagin commented 4 years ago

Describe the bug Nixpkgs patches glibc to look for the locale archive at $LOCALE_PATH. To be more precise, the lookup order is the following:

  1. LOCALE_ARCHIVE_<version>
  2. LOCALE_ARCHIVE
  3. $out/lib/locale/locale-archive (next to libc.so)
  4. /usr/lib/locale/locale-archive

If a program is compiled using Nix and then runs on non-NixOS, it is possible that $LOCALE_ARCHIVE will be unset. It is also very likely in this case that the standard path (the last one) will contain a good locale archive, however, instead of using it, glibc will use the third entry in the list, which is a special locale archive, that contains only one locale: C.utf8, which is, likely, not what the user has configured, and glibc will fallback to POSIX.

The third entry in the list sounds like an okay fallback choice, but wouldn’t it be better to try it only as the last resort?

To Reproduce Steps to reproduce the behaviour:

  1. Write some Haskell code that relies on locales.
  2. Build it with stack with Nix intergration enabled. The program gets linked to glibc from nixpkgs.
  3. Run the program outside of nix-shell with LOCALE_ARCHIVE unset.

The program will work mostly fine, but it will use the locale archive number 3 from the list at the top, so won’t have any locale support, even though there is a perfectly valid locale archive at the standard location.

Expected behavior

If would be better if it used /usr/lib/locale/locale-archive.

Notify maintainers

cc @edolstra @shlevy

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [ glibc ]
# a list of nixos modules affected by the problem
module:
kirelagin commented 4 years ago

For more context: serokell/haskell-with-utf8#7. Seems to be very related: #21820.

edolstra commented 4 years ago

Using /usr/lib/locale/locale-archive is bad for reproducibility. For example, it would cause a (non-sandbox) build to behave differently on a system that has /usr/lib/locale/locale-archive.

kirelagin commented 4 years ago

Are you talking about the use of locale archive during the build? I don’t think it will behave differently, because during the build LOCALE_ARCHIVE is normally set, I believe. But if it can happen that it is not set, the only reasonable choice is to default it to ${pkgs.glibc}/lib/locale/locale-archive (i.e. item 3 in the list).

I think the current version of the patch achieves this, but what I’m proposing is to default it during the build by the way of setting LOCALE_ARCHIVE rather than in the patch. This way the behaviour during the build will be the same (as far, as I can tell) and the behaviour in runtime will be better.

matthewbauer commented 4 years ago

Yeah - since locale-archive is impurely loaded, I wouldn't expect any reproducibility unless you set the LOCALE_ARCHIVE variable. We do a similar things with libGL.

kirelagin commented 4 years ago

So, to reiterate, I propose to swap the order of items 3 and 4 (from my first post) in the patch. The results will be as follows.

During build:

Due to LOCALE_ARCHIVE being always set to something, the order of 3 and 4 does not matter.

In runtime:

kirelagin commented 4 years ago

As a matter of fact, I am not even sure what is the purpose of having current item 3 in the patch at all, as during build it will be in LOCALE_ARCHIVE anyway, and during runtime it is far less likely to exist that the system one, but even if it does, it is not very useful as people don’t normally use C.UTF-8 as their locale.

matthewbauer commented 4 years ago

I think there is a case where you are on NixOS in a pure nix-shell where you want a UTF-8 locale but LOCALE_ARCHIVE is not set. For that you need (3) to be searched. See also https://github.com/NixOS/nixpkgs/pull/61202.

kirelagin commented 4 years ago

Ok, that might be a reason to leave 3 (but still move it to the last place on the list).

However, I think, it should be made illegal to do anything nix-related with LOCALE_ARCHIVE unset. So either the default shellHook should set LOCALE_ARCHIVE to the one from item 4, or nix-shell itself should to it (it gets linked to glibc, so it can find this location, right?).

pickfire commented 4 years ago

Could it be better if nix bundled with full locale, not just C? I believe the size does increase but nix already uses a lot of disk space already (from what I heard), so it probably won't change much adding all the other locales.

matthewbauer commented 4 years ago

Could it be better if nix bundled with full locale, not just C? I believe the size does increase but nix already uses a lot of disk space already (from what I heard), so it probably won't change much adding all the other locales.

Nix itself can use lots of space, but this usually comes from buildtime dependencies, that is things we need to build packages. Runtime dependencies are usually very carefully curated (of course, there are plenty of packages that aren't so optimized). This is important because you need to carry runtime dependencies with you everywhere, while buildtime dependencies are just needed on your builder. For instance, jq is 33.8M:

$ nix path-info -Sh nixpkgs#jq
/nix/store/bpy9adhrnf6yafcq19zvjq5ghqf0n706-jq-1.6-bin    33.8M

while glibcLocales is 209.6M:

$ nix path-info -Sh nixpkgs#glibcLocales
/nix/store/ssk7zb76r9ydy6h6p7kzjsiqq7amcray-glibc-locales-2.31   209.6M
stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info