Compilation fails on CUDA 10.2 arm64v8-a

alborotogarcia commented 3 years ago

Hi Omnisci team!

I get compilation errors building on baremetal arm64 with CUDA, I suspect is due to some linking problem between the c compiler and folly..

I've tried to build with the following setup

With FOLLY On


cmake -DCMAKE_BUILD_TYPE=debug -DENABLE_FOLLY=ON -DENABLE_CUDA=on -DENABLE_AWS_S3=on  -DCMAKE_C_COMPILER=/usr/bin/clang-9 -DCMAKE_CXX_COMPILER=clang++ ..
In file included from /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:46:
/usr/local/include/folly/memory/Arena.h:196:11: error: non-const lvalue reference to type 'SysAllocator<char>' cannot bind to a value of unrelated type 'SysAllocator<void>'
      alloc(), reinterpret_cast<char*>(b), blockGoodAllocSize());
      ^~~~~~~
/usr/local/include/folly/memory/Arena.h:83:5: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::freeBlocks' requested here
freeBlocks();
^
/tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:56:12: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::~Arena' requested here
explicit Arena(size_t min_block_size = static_cast<size_t>(1UL << 32) + kBlockOverhead,
       ^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:332:26: note: passing argument to parameter '__a' here
  deallocate(_Alloc& __a, pointer __p, size_type __n)
                     ^
In file included from /tmp/omniscidb/Fragmenter/UpdelStorage.cpp:28:
In file included from /tmp/omniscidb/Shared/../QueryEngine/Execute.h:29:
In file included from /tmp/omniscidb/Shared/../QueryEngine/GroupByAndAggregate.h:26:
In file included from /tmp/omniscidb/Shared/../QueryEngine/QueryExecutionContext.h:23:
In file included from /tmp/omniscidb/QueryEngine/Rendering/RenderInfo.h:23:
In file included from /tmp/omniscidb/QueryEngine/Descriptors/RowSetMemoryOwner.h:28:
In file included from /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:46:
/usr/local/include/folly/memory/Arena.h:205:31: error: non-const lvalue reference to type 'SysAllocator<char>' cannot bind to a value of unrelated type 'SysAllocator<void>'
  AllocTraits::deallocate(alloc(), reinterpret_cast<char*>(b), size);
                          ^~~~~~~
/usr/local/include/folly/memory/Arena.h:84:5: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::freeLargeBlocks' requested here
freeLargeBlocks();
^
/tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:56:12: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::~Arena' requested here
explicit Arena(size_t min_block_size = static_cast<size_t>(1UL << 32) + kBlockOverhead,
       ^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:332:26: note: passing argument to parameter '__a' here
  deallocate(_Alloc& __a, pointer __p, size_type __n)
                     ^
In file included from /tmp/omniscidb/Fragmenter/UpdelStorage.cpp:28:
In file included from /tmp/omniscidb/Shared/../QueryEngine/Execute.h:29:
In file included from /tmp/omniscidb/Shared/../QueryEngine/GroupByAndAggregate.h:26:
In file included from /tmp/omniscidb/Shared/../QueryEngine/QueryExecutionContext.h:23:
In file included from /tmp/omniscidb/QueryEngine/Rendering/RenderInfo.h:23:
In file included from /tmp/omniscidb/QueryEngine/Descriptors/RowSetMemoryOwner.h:28:
In file included from /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:46:
In file included from /usr/local/include/folly/memory/Arena.h:324:
/usr/local/include/folly/memory/Arena-inl.h:44:30: error: no matching member function for call to 'allocate'
void* mem = AllocTraits::allocate(alloc(), allocSize);
            ~~~~~~~~~~~~~^~~~~~~~
/usr/local/include/folly/memory/Arena.h:111:15: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::allocateSlow' requested here
void* r = allocateSlow(size);
          ^
/tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:62:16: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::allocate' requested here
auto ret = allocate(size);
           ^
/usr/bin/../lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:305:7: note: candidate function not viable: no known conversion from 'SysAllocator<void>' to 'SysAllocator<char> &' for 1st argument
  allocate(_Alloc& __a, size_type __n)
  ^

- And with folly off

cmake -DCMAKE_BUILD_TYPE=debug -DENABLE_FOLLY=OFF -DENABLE_CUDA=on -DENABLE_AWS_S3=on -DCMAKE_C_COMPILER=/usr/bin/clang-9 -DCMAKE_CXX_COMPILER=clang++ ..

[ 44%] Building CXX object QueryEngine/CMakeFiles/QueryEngine.dir/Rendering/os/RenderInfo.cpp.o In file included from /tmp/omniscidb/QueryEngine/UDFCompiler.cpp:17: /tmp/omniscidb/QueryEngine/UDFCompiler.h:28:10: fatal error: 'clang/Driver/Compilation.h' file not found


Can be this a linking problem with llvm/clang?

And could this be compiled using gcc as c compiler?

alexbaden commented 3 years ago

Hi @alborotogarcia ,

First, looks like you need to update Folly. Our current minimum supported version is 2021.01.25, but I have been using the latest version and it seems to do fine on x86.

With folly off, it looks like you are missing clang, or perhaps the clang libraries. If you have that header file, it is possible we are not finding your clang installation. How did you install LLVM/clang?

alborotogarcia commented 3 years ago

@alexbaden Yes I am missing compilation header file.. though I cannot find which lib I am missing

apt list --installed |grep clang

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

clang-9/focal,now 1:9.0.1-12 arm64 [installed]
clang-format-9/focal,now 1:9.0.1-12 arm64 [installed]
clang-tidy-9/focal,now 1:9.0.1-12 arm64 [installed]
clang-tools-9/focal,now 1:9.0.1-12 arm64 [installed]
clangd-9/focal,now 1:9.0.1-12 arm64 [installed]
libclang-common-7-dev/focal,now 1:7.0.1-12 arm64 [installed]
libclang-common-9-dev/focal,now 1:9.0.1-12 arm64 [installed,automatic]
libclang-cpp9/focal,now 1:9.0.1-12 arm64 [installed,automatic]
libclang1-10/focal,now 1:10.0.0-4ubuntu1 arm64 [installed,automatic]
libclang1-9/focal,now 1:9.0.1-12 arm64 [installed,automatic]
apt list --installed |grep llvm

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libllvm10/focal,now 1:10.0.0-4ubuntu1 arm64 [installed,automatic]
libllvm11/focal-updates,now 1:11.0.0-2~ubuntu20.04.1 arm64 [installed,automatic]
libllvm7/focal,now 1:7.0.1-12 arm64 [installed,automatic]
libllvm9/focal,now 1:9.0.1-12 arm64 [installed]
llvm-9-dev/focal,now 1:9.0.1-12 arm64 [installed]
llvm-9-runtime/focal,now 1:9.0.1-12 arm64 [installed]
llvm-9-tools/focal,now 1:9.0.1-12 arm64 [installed]
llvm-9/focal,now 1:9.0.1-12 arm64 [installed]

LLVM_CONFIG is pointing at llvm-9 so does gcc links to gcc-9.

What am I missing?

alborotogarcia commented 3 years ago

I've also tried with clang-8 and llvm-8 due for CUDA 10.2 compatibility and yet I get Compilation.h error header missing.. btw @alexbaden, my faiss version is on 1.7.1

alexbaden commented 3 years ago

Looks like you want libclang-9-dev (or perhaps 8 for Cuda 10.2 compatibility, though we don't officially support < 11 anymore). On x86 ubuntu 18.04 I found Compilation.h in: /usr/lib/llvm-9/include/clang/Driver/Compilation.h after installing the dev package.

alborotogarcia commented 3 years ago

thanks @alexbaden yes you were right! i was missing that one my bad! but now I get Folly Arena Allocation errors errors ..

In file included from /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:46:0,
                 from /tmp/omniscidb/QueryEngine/Descriptors/RowSetMemoryOwner.h:28,
                 from /tmp/omniscidb/QueryEngine/JoinHashTable/BaselineJoinHashTable.h:32,
                 from /tmp/omniscidb/QueryRunner/QueryRunner.h:30,
                 from /tmp/omniscidb/LockMgr/LockMgr.cpp:24:
/usr/local/include/folly/memory/Arena.h: In instantiation of 'folly::Arena<Alloc>::freeBlocks()::<lambda(folly::Arena<Alloc>::Block*)> [with Alloc = SysAllocator<void>]':
/usr/local/include/folly/memory/Arena.h:193:32:   required from 'struct folly::Arena<Alloc>::freeBlocks() [with Alloc = SysAllocator<void>]::<lambda(struct folly::Arena<SysAllocator<void> >::Block*)>'
/usr/local/include/folly/memory/Arena.h:193:5:   required from 'void folly::Arena<Alloc>::freeBlocks() [with Alloc = SysAllocator<void>]'
/usr/local/include/folly/memory/Arena.h:83:15:   required from 'folly::Arena<Alloc>::~Arena() [with Alloc = SysAllocator<void>]'
/tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:59:83:   required from here
/usr/local/include/folly/memory/Arena.h:195:30: error: cannot bind non-const lvalue reference of type 'SysAllocator<char>&' to an rvalue of type 'SysAllocator<char>'
       AllocTraits::deallocate(
       ~~~~~~~~~~~~~~~~~~~~~~~^
           alloc(), reinterpret_cast<char*>(b), blockGoodAllocSize());
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /tmp/omniscidb/QueryEngine/Descriptors/RowSetMemoryOwner.h:28:0,
                 from /tmp/omniscidb/QueryEngine/JoinHashTable/BaselineJoinHashTable.h:32,
                 from /tmp/omniscidb/QueryRunner/QueryRunner.h:30,
                 from /tmp/omniscidb/LockMgr/LockMgr.cpp:24:
/tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:33:13: note:   after user-defined conversion: constexpr SysAllocator<T>::SysAllocator(const SysAllocator<U>&) [with U = void; T = char]
   constexpr SysAllocator(const SysAllocator<U>&) noexcept {}
             ^~~~~~~~~~~~

I have uploaded the full log if this helps https://gist.github.com/alborotogarcia/a12228013b80924f0a13b1c5b3edfead#file-clang-8-log

alexbaden commented 3 years ago

What version of folly do you have?

alborotogarcia commented 3 years ago

Pulled from May 31st release 99fbca1df19fdd21f1b831cad6f50ece94573675 (HEAD -> master, tag: v2021.05.31.00, origin/master, origin/HEAD)

alexbaden commented 3 years ago

Try folly 2021.05.10.00 -- it is possible later versions have broken something.

alborotogarcia commented 3 years ago

Same thing @alexbaden, probably I need to stick to older Folly versions, I find weird that somehow Folly still links to distro gcc In file included from /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:46: /usr/local/include/folly/memory/Arena.h:205:31: error: non-const lvalue reference to type 'SysAllocator<char>' cannot bind to a value of unrelated type 'SysAllocator<void>' AllocTraits::deallocate(alloc(), reinterpret_cast<char*>(b), size); ^~~~~~~ /usr/local/include/folly/memory/Arena.h:84:5: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::freeLargeBlocks' requested here freeLargeBlocks(); ^ /tmp/omniscidb/DataMgr/Allocators/ArenaAllocator.h:56:12: note: in instantiation of member function 'folly::Arena<SysAllocator<void> >::~Arena' requested here explicit Arena(size_t min_block_size = static_cast<size_t>(1UL << 32) + kBlockOverhead, ^ /usr/lib/gcc/aarch64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:332:26: note: passing argument to parameter '__a' here deallocate(_Alloc& __a, pointer __p, size_type __n) ^ when I have already set gcc-7 as default gcc compiler update-alternatives --list gcc /usr/bin/gcc-7

alborotogarcia commented 3 years ago

I have also tried just downloading prebuilt llvm/clang 8.0.1 for arm64 and set it as default And so does the CMakeCache.txt `//No help, variable specified on the command line. CMAKE_CXX_COMPILER:UNINITIALIZED=clang++

//LLVM archiver CMAKE_CXX_COMPILER_AR:FILEPATH=/usr/local/bin/llvm-ar

//Generate index for LLVM archive CMAKE_CXX_COMPILER_RANLIB:FILEPATH=/usr/local/bin/llvm-ranlib

//Flags used by the CXX compiler during all build types. CMAKE_CXX_FLAGS:STRING=

//Flags used by the CXX compiler during DEBUG builds. CMAKE_CXX_FLAGS_DEBUG:STRING=

//Flags used by the CXX compiler during MINSIZEREL builds. CMAKE_CXX_FLAGS_MINSIZEREL:STRING=

//Flags used by the CXX compiler during RELEASE builds. CMAKE_CXX_FLAGS_RELEASE:STRING=

//Flags used by the CXX compiler during RELWITHDEBINFO builds. CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=

//No help, variable specified on the command line. CMAKE_C_COMPILER:UNINITIALIZED=/usr/bin/clang

//LLVM archiver CMAKE_C_COMPILER_AR:FILEPATH=/usr/local/bin/llvm-ar

//Generate index for LLVM archive CMAKE_C_COMPILER_RANLIB:FILEPATH=/usr/local/bin/llvm-ranlib

`

heavyai / heavydb

Compilation fails on CUDA 10.2 arm64v8-a #656