search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.09k stars 4.33k forks source link

Production job failed because of incorrect locale settings #46850

Open makortel opened 2 hours ago

makortel commented 2 hours ago

https://its.cern.ch/jira/browse/CMSPROD-226 shows production job failures (on 16_0_43) caused by incorrect locale settings. Already scram warns

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "C.UTF-8",
    LANG = "C"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

and then later an exception

----- Begin Fatal Exception 23-Oct-2024 05:53:31 UTC-----------------------
An exception of category 'StdException' occurred while
   [0] Processing global begin LuminosityBlock run: 1 luminosityBlock: 7001
   [1] Calling method for module Pythia8GeneratorFilter/'generator'
Exception Message:
A std::exception was thrown.
locale::facet::_S_create_c_locale name not valid
----- End Fatal Exception -------------------------------------------------

is thrown from https://github.com/cms-sw/cmssw/blob/7fe6de81e175c665a6afa3c08205ccb26132f088/GeneratorInterface/EvtGenInterface/plugins/EvtGen/EvtGenInterface.cc#L367 (that was run via ExternalGeneratorFilter).

The error is reproducible locally (in cmssw-el7 container) e.g. by setting LC_ALL=C.UTF-8 before running cmsRun.

I assume we could have also other code that could fail when the locale is set to an incorrect value. I wonder how CMS applications should behave if locale is being set incorrectly. On one hand it feels an invalid locale feels a bit silly reason to fail a job, but on the other hand, how much can we trust on string interpretation if the locale is invalid? Furthermore, I think we have been rather ignorant on locale settings so far, and in any case in the production use case all strings (e.g. job configuration) originate from somewhere else than the worker node (i.e. if we'd really want to handle locales properly, we'd have to propagate the locale from wherever e.g. the configuration file is created; or something).

Should we perhaps enforce some locale in all scram environments?

makortel commented 2 hours ago

assign core

cmsbuild commented 2 hours ago

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 2 hours ago

cms-bot internal usage

cmsbuild commented 2 hours ago

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here