build cpp test fails with GCC at v4.2.0

[natsukium]

Description

When I tried to build cpp_test with gcc, I got the following error. error: explicit specialization in non-namespace scope ‘class ArrowChunkedArrayTest’ error: template-id ‘create_primitive_schema<float>’ in declaration of primary template

[ 97%] Built target _lightgbm
/LightGBM/tests/cpp_tests/test_arrow.cpp:95:13: error: explicit specialization in non-namespace scope ‘class ArrowChunkedArrayTest’
   95 |   template <>
      |             ^
/LightGBM/tests/cpp_tests/test_arrow.cpp:96:46: error: template-id ‘create_primitive_schema<float>’ in declaration of primary template
   96 |   ArrowSchema create_primitive_schema<float>() {
      |                                              ^
make[2]: *** [CMakeFiles/testlightgbm.dir/build.make:76: CMakeFiles/testlightgbm.dir/tests/cpp_tests/test_arrow.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:148: CMakeFiles/testlightgbm.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 98%] Linking CXX executable ../lightgbm
[ 98%] Built target lightgbm
make: *** [Makefile:130: all] Error 2

I'm not familiar with cpp and don't know how to handle it, but it seems that the following template needs to be defined outside of the class. https://github.com/microsoft/LightGBM/blob/4588d648f1007b2a1aa3d8b540cc81e19f348d9f/tests/cpp_tests/test_arrow.cpp#L95-L96

Reproducible example

docker build . -f Dockerfile

with the following Dockerfile

FROM ubuntu:20.04

ENV \
    DEBIAN_FRONTEND=noninteractive \
    LANG=C.UTF-8 \
    LC_ALL=C.UTF-8

RUN apt-get update -y && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        cmake \
        build-essential \
        gcc \
        g++ \
        git \
        libomp-dev && \
    rm -rf /var/lib/apt/lists/*

RUN git clone \
        --recursive \
        --branch stable \
        --depth 1 \
        https://github.com/Microsoft/LightGBM && \
    mkdir LightGBM/build && \
    cd LightGBM/build && \
    cmake .. -DBUILD_CPP_TEST=ON && \
    make -j12 && \
    make install && \
    cd "${HOME}" && \
    rm -rf LightGBM

ENTRYPOINT ["lightgbm"]

Environment info

LightGBM version or commit hash: 3405ee82b98ab4dd26706d98b7b77e3f079d2b2c or later

Command(s) you used to install LightGBM

cd LightGBM/build && cmake .. -DBUILD_CPP_TEST=ON && make -j

Additional Comments

[jameslamb]

Thanks very much for the excellent report, and sorry for the inconvenience.

This is making me realize we don't actually run the c++ tests with gcc in continuous integration. Just:

clang-17 on Ubuntu 22.04:

https://github.com/microsoft/LightGBM/blob/4588d648f1007b2a1aa3d8b540cc81e19f348d9f/.vsts-ci.yml#L126-L128

AppleClang (unsure of the version) on macOS 11

https://github.com/microsoft/LightGBM/blob/4588d648f1007b2a1aa3d8b540cc81e19f348d9f/.vsts-ci.yml#L235-L238

MSVC 2019 on Windows

https://github.com/microsoft/LightGBM/blob/4588d648f1007b2a1aa3d8b540cc81e19f348d9f/.vsts-ci.yml#L275-L276

[jameslamb]

@borchero is this something you could help with? I think the fix would be:

1. add a cpp_test CI job to the Linux matrix at https://github.com/microsoft/LightGBM/blob/4588d648f1007b2a1aa3d8b540cc81e19f348d9f/.vsts-ci.yml#L30
2. confirm that it reproduces the error reported here
3. push additional commits fixing the issue

At the current low level of activity in this project, I think we can afford one more cpp_test CI job on Azure DevOps, in exchange for being able to catch these issues in the future.

[borchero]

I'll have a look at this!

[borchero]

I can't replicate the issue locally, unfortunately, let's see if #6315 helps.

[jameslamb]

I ran into this today investigating #4331.

On my M2 mac, latest master here.

brew install gcc-13

cmake \
    -B build \
    -S . \
    -DBUILD_CPP_TEST=ON \
    -DCMAKE_C_COMPILER='gcc-13' \
    -DCMAKE_CXX_COMPILER='g++-13'

cmake \
    --build build \
    --target testlightgbm \
    -j4

Configure logs.

-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/homebrew/bin/gcc-13 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/homebrew/bin/g++-13 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Failed
-- Could NOT find GTest (missing: GTest_DIR)
-- Did not find Google Test in the system root. Fetching Google Test now...
-- Found Python3: /Users/jlamb/miniforge3/envs/pydistcheck-dev/bin/python3.12 (found version "3.12.3") found components: Interpreter 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Configuring done (3.8s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/jlamb/repos/LightGBM/build

End of the build logs.

[ 85%] Building CXX object CMakeFiles/testlightgbm.dir/tests/cpp_tests/test_chunked_array.cpp.o
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:159:13: error: explicit specialization in non-namespace scope 'class ArrowChunkedArrayTest'
  159 |   template <>
      |             ^
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:160:15: error: template-id 'create_primitive_schema<float>' in declaration of primary template
  160 |   ArrowSchema create_primitive_schema<float>() {
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:174:13: error: explicit specialization in non-namespace scope 'class ArrowChunkedArrayTest'
  174 |   template <>
      |             ^
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:175:15: error: template-id 'create_primitive_schema<bool>' in declaration of primary template
  175 |   ArrowSchema create_primitive_schema<bool>() {
...
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:175:15: error: 'ArrowSchema ArrowChunkedArrayTest::create_primitive_schema()' cannot be overloaded with 'ArrowSchema ArrowChunkedArrayTest::create_primitive_schema()'
/Users/jlamb/repos/LightGBM/tests/cpp_tests/test_arrow.cpp:160:15: note: previous declaration 'ArrowSchema ArrowChunkedArrayTest::create_primitive_schema()'
  160 |   ArrowSchema create_primitive_schema<float>() {
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: note: unrecognized command-line option '-Wno-unused-private-field' may have been intended to silence earlier diagnostics
make[3]: *** [CMakeFiles/testlightgbm.dir/tests/cpp_tests/test_arrow.cpp.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [CMakeFiles/testlightgbm.dir/all] Error 2
make[1]: *** [CMakeFiles/testlightgbm.dir/rule] Error 2
make: *** [testlightgbm] Error 

A patch like this fixed it for me:

diff --git a/tests/cpp_tests/test_arrow.cpp b/tests/cpp_tests/test_arrow.cpp
index c2dbd6ce..29bff7dc 100644
--- a/tests/cpp_tests/test_arrow.cpp
+++ b/tests/cpp_tests/test_arrow.cpp
@@ -151,13 +151,7 @@ class ArrowChunkedArrayTest : public testing::Test {

   /* ------------------------------------- SCHEMA CREATION ------------------------------------- */

-  template <typename T>
-  ArrowSchema create_primitive_schema() {
-    std::logic_error("not implemented");
-  }
-
-  template <>
-  ArrowSchema create_primitive_schema<float>() {
+  ArrowSchema create_primitive_schema_float() {
     ArrowSchema schema;
     schema.format = "f";
     schema.name = nullptr;
@@ -171,8 +165,7 @@ class ArrowChunkedArrayTest : public testing::Test {
     return schema;
   }

-  template <>
-  ArrowSchema create_primitive_schema<bool>() {
+  ArrowSchema create_primitive_schema_bool() {
     ArrowSchema schema;
     schema.format = "b";
     schema.name = nullptr;
@@ -213,7 +206,7 @@ class ArrowChunkedArrayTest : public testing::Test {
 /* --------------------------------------------------------------------------------------------- */

 TEST_F(ArrowChunkedArrayTest, GetLength) {
-  auto schema = create_primitive_schema<float>();
+  auto schema = create_primitive_schema_float();

   std::vector<float> dat1 = {1, 2};
   auto arr1 = create_primitive_array(dat1);
@@ -241,8 +234,8 @@ TEST_F(ArrowChunkedArrayTest, GetColumns) {
   std::vector<ArrowArray*> arrs = {&arr1, &arr2};
   auto arr = created_nested_array(arrs);

-  auto schema1 = create_primitive_schema<float>();
-  auto schema2 = create_primitive_schema<float>();
+  auto schema1 = create_primitive_schema_float();
+  auto schema2 = create_primitive_schema_float();
   std::vector<ArrowSchema*> schemas = {&schema1, &schema2};
   auto schema = create_nested_schema(schemas);

@@ -266,7 +259,7 @@ TEST_F(ArrowChunkedArrayTest, IteratorArithmetic) {
   auto arr2 = create_primitive_array(dat2);
   std::vector<float> dat3 = {7};
   auto arr3 = create_primitive_array(dat3);
-  auto schema = create_primitive_schema<float>();
+  auto schema = create_primitive_schema_float();

   ArrowArray arrs[3] = {arr1, arr2, arr3};
   ArrowChunkedArray ca(3, arrs, &schema);
@@ -302,7 +295,7 @@ TEST_F(ArrowChunkedArrayTest, BooleanIterator) {
   auto arr1 = create_primitive_array(dat1, 0, {2});
   std::vector<bool> dat2 = {false, false, false, false, true, true, true, true, false, true};
   auto arr2 = create_primitive_array(dat2, 1);
-  auto schema = create_primitive_schema<bool>();
+  auto schema = create_primitive_schema_bool();

   ArrowArray arrs[2] = {arr1, arr2};
   ArrowChunkedArray ca(2, arrs, &schema);
@@ -328,7 +321,7 @@ TEST_F(ArrowChunkedArrayTest, BooleanIterator) {
 TEST_F(ArrowChunkedArrayTest, OffsetAndValidity) {
   std::vector<float> dat = {0, 1, 2, 3, 4, 5, 6};
   auto arr = create_primitive_array(dat, 2, {2, 3});
-  auto schema = create_primitive_schema<float>();
+  auto schema = create_primitive_schema_float();
   ArrowChunkedArray ca(1, &arr, &schema);

   auto it = ca.begin<double>();

@borchero could you try that on #6315?

[StrikerRUS]

Just faced this issue with MinGW (gcc-14) on Windows.

Upstream bug in gcc: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85282

Another possible workaround: https://cplusplus.com/forum/beginner/188847/