Problem with wrong locale Setting of LC_CTYPE

[URZ-HD]

Hi, currently a lot of users has problems mit wrong locale settings. Their client sets the "LC_CTYPE" paramater to an invalid value, e.g. LC_CTYPE=utf8 (instead of LC_CTYPE=de_DE.UTF-8).

As a result ws_allocate shows:

> ws_allocate localetest --debug
Debug: secondary group XXX
Debug: secondary group XXX
debug: primarygroup=XXX
debug: no filesystem given, searching...
debug: searching work
debug: not allocatable, skipping 
debug: searching gpfs
debug: fallback, using global default, ending search
debug: find_valid_fs:work
debug: find_valid_fs, granted
debug: find_valid_fs:gpfs
debug: find_valid_fs, granted
debug: moved default filesystem to front:gpfs
debug: searching valid filesystems gpfs
debug: searching valid filesystems work
Info: creating workspace.
Error: could not create workspace directory!

In the source code it is mentioned that the code will not work with wrong/invalid locale settings. But as i understand this should force the correct setting?

    // we only support C locale, if the used local is not installed on the system
    // ws_allocate fails
    setenv("LANG","C",1);
    std::setlocale(LC_ALL, "C");
    std::locale::global(std::locale("C"));

So i'm not sure if it is a bug. But even if its not, is it possible to make the error message more clear to the user ? Than it can easily be solved by setting a valid value for LC_CTYPE.

thanks.

[holgerBerger]

hmm yes, that should have been solved.

I can not see that effect on Ubuntu 18.4, but I can see it on a redhat 7.9 installation, strace shows that a invalid path is to load locales is generated.

I can not build on that machine at the moment, can you try to set in addition std::setlocale(LC_CTYPE, "C"); and see if that helps?

In my understanding LC_ALL should set... well... all...

[URZ-HD]

Hi, unfortunately this does not fix the issue:

Changed the code:

    // we only support C locale, if the used local is not installed on the system
    // ws_allocate fails
    setenv("LANG","C",1);
    std::setlocale(LC_ALL, "C");
    std::setlocale(LC_CTYPE, "C");
    std::locale::global(std::locale("C"));

Error message is still:

Info: creating workspace.
Error: could not create workspace directory!

Our system is based on:

CentOS Linux release 7.7.1908 (Core)
Linux 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

locale Environment:

> printenv | grep LC
LC_PAPER=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_MONETARY=de_DE.UTF-8
LC_NUMERIC=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_CTYPE=utf8
LC_TIME=de_DE.UTF-8
LC_NAME=de_DE.UTF-8

[URZ-HD]

I've traced the process:

[...]
write(2, "Info: creating workspace.", 25Info: creating workspace.) = 25
write(2, "\n", 1
)                       = 1
time(NULL)                              = 1612479637
setresuid(-1, 0, -1)                    = 0
umask(077)                              = 02
stat("/gpfs/bwfor/work/ws/root-localetest2", 0x7ffda04b3530) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106075056, ...}) = 0
mmap(NULL, 106075056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b3748c8d000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b3748373000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x2b3748373000, 4096)            = 0
open("/usr/lib/locale/utf8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)   <-----------
setresuid(-1, 85, -1)                   = 0
write(2, "Error: could not create workspac"..., 44Error: could not create workspace directory!) = 44
write(2, "\n", 1
)                       = 1
exit_group(-1)                          = ?
+++ exited with 255 +++

So you can see, that the current value of "utf8" is used, instead of "C"

[holgerBerger]

can you try setenv("LC_CTYPE,"C",1); in addition?

seems to happen in fs::create_directories(wsdir); in src/ws.cpp

that line could be replaced with mkdir(wsdir.c_str(), 0700);

not sure if that helps, may be you can try both attempts? (mkdir would require some new errror checking)

[URZ-HD]

Hi, adding

setenv("LC_CTYPE,"C",1);

solves the problem!. So the working code looks like this:

 setenv("LANG","C",1);
    setenv("LC_CTYPE","C",1);
    std::setlocale(LC_ALL, "C");
    std::locale::global(std::locale("C"));

No need for std::setlocale(LC_CTYPE, "C");

[URZ-HD]

The same fix has to be applied to ws_release, because of:

> ws_release foobar2
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Aborted

Adding setenv("LC_CTYPE","C",1); fix this issue, too.

[holgerBerger]

which boost version is used in that build?

[URZ-HD]

boost-1.53.0-27.el7.x86_64

[URZ-HD]

The problem still occurs if users have invalid locale Settings, e.g.:

LC_ALL=enUS.UTF-8  (which is invalid, should be "en_US.UTF-8")
LANG=en_US.UTF-8

No other locale variables set After changing LC_ALL to a valid value, everything works as expected.

So I changed

std::setlocale(LC_ALL, "C"); into setenv("LC_ALL","C",1);

This fixed the problem. But i'm not sure if we need to set all available locale Variables explicitly ?

[holgerBerger]

hmm can not reproduce it with centos 8.

does

diff --git a/src/ws_allocate.cpp b/src/ws_allocate.cpp
index dcc6467..d6c1f9b 100644
--- a/src/ws_allocate.cpp
+++ b/src/ws_allocate.cpp
@@ -43,6 +43,7 @@

 #include <boost/program_options.hpp>
 #include <boost/regex.hpp>
+#include <boost/filesystem.hpp>

 #ifndef SETUID
 #include <sys/capability.h>
@@ -212,6 +213,7 @@ int main(int argc, char **argv) {
     setenv("LC_CTYPE","C",1);
     std::setlocale(LC_ALL, "C");
     std::locale::global(std::locale("C"));
+       boost::filesystem::path::imbue(std::locale());

     // read config (for dbuid, reminder and duration default only)
     YAML::Node config;

change anything?

I wonder if that happens in the boost version.

[URZ-HD]

Hi, i've done the mentioned changes in ws_allocate, but the problem still persists. The only way that worked so far, is to use "setenv("LC_ALL","C",1);" instead of "std::setlocale(LC_ALL, "C");"