[clang-repl] JIT linkage issues with extern templates

[jeaye]

Overview

Hey folks! I've found that the clang interpreter on the latest 19 build cannot link to extern templates definitions. I have verified that this bug also existed in release/18.x, so it's not new to 19.

Pinging @vgvassilev for vis, since I originally reported this to him with a more complex repro case.

Test case

This requires linking to an extern template which is defined either within the current process or in a loaded shared library. In my test case, I follow the existing pattern of shared object loading to build a shared object containing our defined function. You can see the patch to add the test here:

diff --git a/clang/test/Interpreter/Inputs/extern-template.cpp b/clang/test/Interpreter/Inputs/extern-template.cpp
new file mode 100644
index 000000000000..e08263a96c8f
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.cpp
@@ -0,0 +1,3 @@
+#include <cstdio>
+
+char foo(char c) { printf("good %c", c); return c; }
diff --git a/clang/test/Interpreter/Inputs/extern-template.hpp b/clang/test/Interpreter/Inputs/extern-template.hpp
new file mode 100644
index 000000000000..ac067a202dc8
--- /dev/null
+++ b/clang/test/Interpreter/Inputs/extern-template.hpp
@@ -0,0 +1,5 @@
+#pragma once
+
+template <typename T> T foo(T t) { return t; }
+
+extern template char foo(char c);
diff --git a/clang/test/Interpreter/extern-template.cpp b/clang/test/Interpreter/extern-template.cpp
new file mode 100644
index 000000000000..edb586285bdf
--- /dev/null
+++ b/clang/test/Interpreter/extern-template.cpp
@@ -0,0 +1,16 @@
+// REQUIRES: host-supports-jit, x86_64-linux
+
+// To generate libextern-template.so :
+// clang -xc++ -o Inputs/libextern-template.so -fPIC -shared Inputs/extern-template.cpp
+
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs | FileCheck %s
+// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs -Xcc -O2 | FileCheck %s
+
+extern "C" int printf(const char* format, ...);
+
+#include "extern-template.hpp"
+
+%lib libextern-template.so
+
+foo('a');
+// CHECK: good

The output of running this test (after having compiled libextern-template.so):

❯ ./bin/llvm-lit -j1 -v ../clang/test/Interpreter/extern-template.cpp
llvm-lit: /home/jeaye/projects/third-party/llvm-project/llvm/utils/lit/lit/llvm/config.py:509: note: using clang: /home/jeaye/projects/third-party/llvm-project/build/bin/clang
-- Testing: 1 tests, 1 workers --
FAIL: Clang :: Interpreter/extern-template.cpp (1 of 1)
******************** TEST 'Clang :: Interpreter/extern-template.cpp' FAILED ********************
Exit Code: 1

...

JIT session error: Symbols not found: [ _Z3fooIcET_S0_ ]
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15, $.incr_module_15.__inits.0 }) }
error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15 }) }
/home/jeaye/projects/third-party/llvm-project/clang/test/Interpreter/extern-template.cpp:16:11: error: CHECK: expected string not found in input
// CHECK: good

...

********************
********************
Failed Tests (1):
  Clang :: Interpreter/extern-template.cpp
Testing Time: 0.05s
Total Discovered Tests: 1
  Failed: 1 (100.00%)

Additional notes

In the Inputs/extern-template.hpp file, we have this line:

extern template char foo(char c);

If it is instead changed to this, the test will pass:

char foo(char c);

Real world example

My test case is reduced down from a real world use case I originally noticed in the fmt library: https://github.com/fmtlib/fmt/blob/b61c8c3d23b7e6fdf9d44593877dba1c8a291be1/include/fmt/format.h#L4283

[llvmbot]

@llvm/issue-subscribers-orcjit

Author: Jeaye Wilkerson (jeaye)

## Overview Hey folks! I've found that the clang interpreter on the latest 19 build cannot link to extern templates definitions. I have verified that this bug also existed in `release/18.x`, so it's not new to 19. Pinging @vgvassilev for vis, since I originally reported this to him with a more complex repro case. ## Test case This requires linking to an extern template which is defined either within the current process or in a loaded shared library. In my test case, I follow the existing pattern of shared object loading to build a shared object containing our defined function. You can see the patch to add the test here: ```diff diff --git a/clang/test/Interpreter/Inputs/extern-template.cpp b/clang/test/Interpreter/Inputs/extern-template.cpp new file mode 100644 index 000000000000..e08263a96c8f --- /dev/null +++ b/clang/test/Interpreter/Inputs/extern-template.cpp @@ -0,0 +1,3 @@ +#include <cstdio> + +char foo(char c) { printf("good %c", c); return c; } diff --git a/clang/test/Interpreter/Inputs/extern-template.hpp b/clang/test/Interpreter/Inputs/extern-template.hpp new file mode 100644 index 000000000000..ac067a202dc8 --- /dev/null +++ b/clang/test/Interpreter/Inputs/extern-template.hpp @@ -0,0 +1,5 @@ +#pragma once + +template <typename T> T foo(T t) { return t; } + +extern template char foo(char c); diff --git a/clang/test/Interpreter/extern-template.cpp b/clang/test/Interpreter/extern-template.cpp new file mode 100644 index 000000000000..edb586285bdf --- /dev/null +++ b/clang/test/Interpreter/extern-template.cpp @@ -0,0 +1,16 @@ +// REQUIRES: host-supports-jit, x86_64-linux + +// To generate libextern-template.so : +// clang -xc++ -o Inputs/libextern-template.so -fPIC -shared Inputs/extern-template.cpp + +// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs | FileCheck %s +// RUN: cat %s | env LD_LIBRARY_PATH=%S/Inputs:$LD_LIBRARY_PATH clang-repl -Xcc -I%S/Inputs -Xcc -O2 | FileCheck %s + +extern "C" int printf(const char* format, ...); + +#include "extern-template.hpp" + +%lib libextern-template.so + +foo('a'); +// CHECK: good ``` The output of running this test (after having compiled `libextern-template.so`): ``` ❯ ./bin/llvm-lit -j1 -v ../clang/test/Interpreter/extern-template.cpp llvm-lit: /home/jeaye/projects/third-party/llvm-project/llvm/utils/lit/lit/llvm/config.py:509: note: using clang: /home/jeaye/projects/third-party/llvm-project/build/bin/clang -- Testing: 1 tests, 1 workers -- FAIL: Clang :: Interpreter/extern-template.cpp (1 of 1) ******************** TEST 'Clang :: Interpreter/extern-template.cpp' FAILED ******************** Exit Code: 1 ... JIT session error: Symbols not found: [ _Z3fooIcET_S0_ ] error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15, $.incr_module_15.__inits.0 }) } error: Failed to materialize symbols: { (main, { __orc_init_func.incr_module_15 }) } /home/jeaye/projects/third-party/llvm-project/clang/test/Interpreter/extern-template.cpp:16:11: error: CHECK: expected string not found in input // CHECK: good ... ******************** ******************** Failed Tests (1): Clang :: Interpreter/extern-template.cpp Testing Time: 0.05s Total Discovered Tests: 1 Failed: 1 (100.00%) ``` ## Additional notes In the `Inputs/extern-template.hpp` file, we have this line: ```cpp extern template char foo(char c); ``` If it is instead changed to this, the test will pass: ```cpp char foo(char c); ``` ## Real world example My test case is reduced down from a real world use case I originally noticed in the fmt library: https://github.com/fmtlib/fmt/blob/b61c8c3d23b7e6fdf9d44593877dba1c8a291be1/include/fmt/format.h#L4283