search

carstenbauer / ThreadPinning.jl

Readily pin Julia threads to CPU-threads
https://carstenbauer.github.io/ThreadPinning.jl/
MIT License
106 stars 7 forks source link

Precompilation/importing failure in case of non-english locale #58

Closed aw32 closed 1 year ago

aw32 commented 1 year ago

ThreadPinning.jl fails if the locale is not english. With LANG=de_DE.UTF-8 the following output is printed:

┌ Warning: Number of online CPUs (0) doesn't match Sys.CPU_THREADS (128).
└ @ ThreadPinning depottest/packages/ThreadPinning/bxPtn/src/sysinfo.jl:106
┌ Warning: Number of online CPUs (0) doesn't match Sys.CPU_THREADS (128).
└ @ ThreadPinning depottest/packages/ThreadPinning/bxPtn/src/sysinfo.jl:106
ERROR: LoadError: ArgumentError: Inavlid CPU ID encountered. See `cpuids_all()` for all valid CPU IDs on the system.
Stacktrace:
 [1] pinthread(cpuid::Int64; warn::Bool)
   @ ThreadPinning depottest/packages/ThreadPinning/bxPtn/src/pinning.jl:8
 [2] pinthread
   @ depottest/packages/ThreadPinning/bxPtn/src/pinning.jl:5 [inlined]
 [3] macro expansion
   @ depottest/packages/ThreadPinning/bxPtn/src/ThreadPinning.jl:87 [inlined]
 [4] top-level scope
   @ depottest/packages/SnoopPrecompile/1XXT1/src/SnoopPrecompile.jl:62
 [5] include
   @ ./Base.jl:419 [inlined]
 [6] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base ./loading.jl:1554
 [7] top-level scope
   @ stdin:1
in expression starting at depottest/packages/ThreadPinning/bxPtn/src/ThreadPinning.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile ThreadPinning [811555cd-349b-4f26-b7bc-1f208b848042] to depottest/compiled/v1.8/ThreadPinning/jl_JA49AD.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
   @ Base ./loading.jl:1707
 [3] compilecache
   @ ./loading.jl:1651 [inlined]
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1337
 [5] _require_prelocked(uuidkey::Base.PkgId)
   @ Base ./loading.jl:1200
 [6] macro expansion
   @ ./loading.jl:1180 [inlined]
 [7] macro expansion
   @ ./lock.jl:223 [inlined]
 [8] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144

When running Julia with LANG=en it works. Tested with Julia 1.8.3 + ThreadPinning v0.7.2 and ThreadPinning current main d2f2bb2.

carstenbauer commented 1 year ago

Unfortunately, I can't reproduce with the same Julia and package versions. Works just fine for me (I'm using a clean Julia depot to make sure that the precompilation cache is empty).

➜  bauerc@n2fpga19 temp
$ export JULIA_DEPOT_PATH=/scratch/pc2-mitarbeiter/bauerc/.juliatest

➜  bauerc@n2fpga19 temp
$ export LANG=de_DE.UTF-8

➜  bauerc@n2fpga19 temp
$ julia. -q
(temp) pkg> add ThreadPinning
  Installing known registries into `/scratch/pc2-mitarbeiter/bauerc/.juliatest`
    Updating registry at `/scratch/pc2-mitarbeiter/bauerc/.juliatest/registries/General.toml`
   Resolving package versions...
   Installed SnoopPrecompile ─ v1.0.3
   Installed Preferences ───── v1.3.0
   Installed ThreadPinning ─── v0.7.2
  No Changes to `~/temp/Project.toml`
  No Changes to `~/temp/Manifest.toml`
Precompiling project...
  6 dependencies successfully precompiled in 4 seconds

julia> using ThreadPinning

julia> threadinfo()

| 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,
  16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,
  32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,
  48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63 |
| 64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,
  80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,
  96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,
  112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127 |

# = Julia thread, | = Socket seperator

Julia threads: 1
├ Occupied CPU-threads: 1
└ Mapping (Thread => CPUID): 1 => 112,
carstenbauer commented 1 year ago

What do you get for lscpu --all --extended with non-english LANG? That's the only thing I can think of right now that could, in principle, be language dependent and cause issues. (It isn't on my system though...)

aw32 commented 1 year ago 
$ lscpu --all --extended
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
0   0    0      0    0:0:0:0       ja     3529,0520 1500,0000
1   0    0      1    1:1:1:0       ja     3529,0520 1500,0000
2   0    0      2    2:2:2:0       ja     3529,0520 1500,0000
3   0    0      3    3:3:3:0       ja     3529,0520 1500,0000
4   0    0      4    4:4:4:0       ja     3529,0520 1500,0000
5   0    0      5    5:5:5:0       ja     3529,0520 1500,0000
6   0    0      6    6:6:6:0       ja     3529,0520 1500,0000
7   0    0      7    7:7:7:0       ja     3529,0520 1500,0000
8   0    0      8    8:8:8:1       ja     3529,0520 1500,0000
9   0    0      9    9:9:9:1       ja     3529,0520 1500,0000
10  0    0      10   10:10:10:1    ja     3529,0520 1500,0000
11  0    0      11   11:11:11:1    ja     3529,0520 1500,0000
12  0    0      12   12:12:12:1    ja     3529,0520 1500,0000
13  0    0      13   13:13:13:1    ja     3529,0520 1500,0000
14  0    0      14   14:14:14:1    ja     3529,0520 1500,0000
15  0    0      15   15:15:15:1    ja     3529,0520 1500,0000
16  1    0      16   16:16:16:2    ja     3529,0520 1500,0000
17  1    0      17   17:17:17:2    ja     3529,0520 1500,0000
18  1    0      18   18:18:18:2    ja     3529,0520 1500,0000
19  1    0      19   19:19:19:2    ja     3529,0520 1500,0000
20  1    0      20   20:20:20:2    ja     3529,0520 1500,0000
21  1    0      21   21:21:21:2    ja     3529,0520 1500,0000
22  1    0      22   22:22:22:2    ja     3529,0520 1500,0000
23  1    0      23   23:23:23:2    ja     3529,0520 1500,0000
24  1    0      24   24:24:24:3    ja     3529,0520 1500,0000
25  1    0      25   25:25:25:3    ja     3529,0520 1500,0000
26  1    0      26   26:26:26:3    ja     3529,0520 1500,0000
27  1    0      27   27:27:27:3    ja     3529,0520 1500,0000
28  1    0      28   28:28:28:3    ja     3529,0520 1500,0000
29  1    0      29   29:29:29:3    ja     3529,0520 1500,0000
30  1    0      30   30:30:30:3    ja     3529,0520 1500,0000
31  1    0      31   31:31:31:3    ja     3529,0520 1500,0000
32  2    0      32   32:32:32:4    ja     3529,0520 1500,0000
33  2    0      33   33:33:33:4    ja     3529,0520 1500,0000
34  2    0      34   34:34:34:4    ja     3529,0520 1500,0000
35  2    0      35   35:35:35:4    ja     3529,0520 1500,0000
36  2    0      36   36:36:36:4    ja     3529,0520 1500,0000
37  2    0      37   37:37:37:4    ja     3529,0520 1500,0000
38  2    0      38   38:38:38:4    ja     3529,0520 1500,0000
39  2    0      39   39:39:39:4    ja     3529,0520 1500,0000
40  2    0      40   40:40:40:5    ja     3529,0520 1500,0000
41  2    0      41   41:41:41:5    ja     3529,0520 1500,0000
42  2    0      42   42:42:42:5    ja     3529,0520 1500,0000
43  2    0      43   43:43:43:5    ja     3529,0520 1500,0000
44  2    0      44   44:44:44:5    ja     3529,0520 1500,0000
45  2    0      45   45:45:45:5    ja     3529,0520 1500,0000
46  2    0      46   46:46:46:5    ja     3529,0520 1500,0000
47  2    0      47   47:47:47:5    ja     3529,0520 1500,0000
48  3    0      48   48:48:48:6    ja     3529,0520 1500,0000
49  3    0      49   49:49:49:6    ja     3529,0520 1500,0000
50  3    0      50   50:50:50:6    ja     3529,0520 1500,0000
51  3    0      51   51:51:51:6    ja     3529,0520 1500,0000
52  3    0      52   52:52:52:6    ja     3529,0520 1500,0000
53  3    0      53   53:53:53:6    ja     3529,0520 1500,0000
54  3    0      54   54:54:54:6    ja     3529,0520 1500,0000
55  3    0      55   55:55:55:6    ja     3529,0520 1500,0000
56  3    0      56   56:56:56:7    ja     3529,0520 1500,0000
57  3    0      57   57:57:57:7    ja     3529,0520 1500,0000
58  3    0      58   58:58:58:7    ja     3529,0520 1500,0000
59  3    0      59   59:59:59:7    ja     3529,0520 1500,0000
60  3    0      60   60:60:60:7    ja     3529,0520 1500,0000
61  3    0      61   61:61:61:7    ja     3529,0520 1500,0000
62  3    0      62   62:62:62:7    ja     3529,0520 1500,0000
63  3    0      63   63:63:63:7    ja     3529,0520 1500,0000
64  0    0      0    0:0:0:0       ja     3529,0520 1500,0000
65  0    0      1    1:1:1:0       ja     3529,0520 1500,0000
66  0    0      2    2:2:2:0       ja     3529,0520 1500,0000
67  0    0      3    3:3:3:0       ja     3529,0520 1500,0000
68  0    0      4    4:4:4:0       ja     3529,0520 1500,0000
69  0    0      5    5:5:5:0       ja     3529,0520 1500,0000
70  0    0      6    6:6:6:0       ja     3529,0520 1500,0000
71  0    0      7    7:7:7:0       ja     3529,0520 1500,0000
72  0    0      8    8:8:8:1       ja     3529,0520 1500,0000
73  0    0      9    9:9:9:1       ja     3529,0520 1500,0000
74  0    0      10   10:10:10:1    ja     3529,0520 1500,0000
75  0    0      11   11:11:11:1    ja     3529,0520 1500,0000
76  0    0      12   12:12:12:1    ja     3529,0520 1500,0000
77  0    0      13   13:13:13:1    ja     3529,0520 1500,0000
78  0    0      14   14:14:14:1    ja     3529,0520 1500,0000
79  0    0      15   15:15:15:1    ja     3529,0520 1500,0000
80  1    0      16   16:16:16:2    ja     3529,0520 1500,0000
81  1    0      17   17:17:17:2    ja     3529,0520 1500,0000
82  1    0      18   18:18:18:2    ja     3529,0520 1500,0000
83  1    0      19   19:19:19:2    ja     3529,0520 1500,0000
84  1    0      20   20:20:20:2    ja     3529,0520 1500,0000
85  1    0      21   21:21:21:2    ja     3529,0520 1500,0000
86  1    0      22   22:22:22:2    ja     3529,0520 1500,0000
87  1    0      23   23:23:23:2    ja     3529,0520 1500,0000
88  1    0      24   24:24:24:3    ja     3529,0520 1500,0000
89  1    0      25   25:25:25:3    ja     3529,0520 1500,0000
90  1    0      26   26:26:26:3    ja     3529,0520 1500,0000
91  1    0      27   27:27:27:3    ja     3529,0520 1500,0000
92  1    0      28   28:28:28:3    ja     3529,0520 1500,0000
93  1    0      29   29:29:29:3    ja     3529,0520 1500,0000
94  1    0      30   30:30:30:3    ja     3529,0520 1500,0000
95  1    0      31   31:31:31:3    ja     3529,0520 1500,0000
96  2    0      32   32:32:32:4    ja     3529,0520 1500,0000
97  2    0      33   33:33:33:4    ja     3529,0520 1500,0000
98  2    0      34   34:34:34:4    ja     3529,0520 1500,0000
99  2    0      35   35:35:35:4    ja     3529,0520 1500,0000
100 2    0      36   36:36:36:4    ja     3529,0520 1500,0000
101 2    0      37   37:37:37:4    ja     3529,0520 1500,0000
102 2    0      38   38:38:38:4    ja     3529,0520 1500,0000
103 2    0      39   39:39:39:4    ja     3529,0520 1500,0000
104 2    0      40   40:40:40:5    ja     3529,0520 1500,0000
105 2    0      41   41:41:41:5    ja     3529,0520 1500,0000
106 2    0      42   42:42:42:5    ja     3529,0520 1500,0000
107 2    0      43   43:43:43:5    ja     3529,0520 1500,0000
108 2    0      44   44:44:44:5    ja     3529,0520 1500,0000
109 2    0      45   45:45:45:5    ja     3529,0520 1500,0000
110 2    0      46   46:46:46:5    ja     3529,0520 1500,0000
111 2    0      47   47:47:47:5    ja     3529,0520 1500,0000
112 3    0      48   48:48:48:6    ja     3529,0520 1500,0000
113 3    0      49   49:49:49:6    ja     3529,0520 1500,0000
114 3    0      50   50:50:50:6    ja     3529,0520 1500,0000
115 3    0      51   51:51:51:6    ja     3529,0520 1500,0000
116 3    0      52   52:52:52:6    ja     3529,0520 1500,0000
117 3    0      53   53:53:53:6    ja     3529,0520 1500,0000
118 3    0      54   54:54:54:6    ja     3529,0520 1500,0000
119 3    0      55   55:55:55:6    ja     3529,0520 1500,0000
120 3    0      56   56:56:56:7    ja     3529,0520 1500,0000
121 3    0      57   57:57:57:7    ja     3529,0520 1500,0000
122 3    0      58   58:58:58:7    ja     3529,0520 1500,0000
123 3    0      59   59:59:59:7    ja     3529,0520 1500,0000
124 3    0      60   60:60:60:7    ja     3529,0520 1500,0000
125 3    0      61   61:61:61:7    ja     3529,0520 1500,0000
126 3    0      62   62:62:62:7    ja     3529,0520 1500,0000
127 3    0      63   63:63:63:7    ja     3529,0520 1500,0000
$ echo $LANG
de_DE.UTF-8
carstenbauer commented 1 year ago

Interesting! The "ja"s are likely the issue. The line that needs fixing is https://github.com/carstenbauer/ThreadPinning.jl/blob/main/src/sysinfo.jl#L103. A very pragmatic solution would be to just also look for "ja". But of course, this would only make things work for EN and DE. Given that the online column is irrelevant on virtually all systems (I only added support for it because FUGAKU has offline cpus....) we could also just say that if we can't find any "no"s than we assume that all CPUs are online. So, essentially only supporting offline CPUs on EN systems.

Feel free to make a PR for this! Otherwise, I'll do it when I find time for it.

aw32 commented 1 year ago

Now it works. Thanks!