Update to LLVM 6

[WebDrake]

This patch updates the LDC 1.10.0 snap to use the latest version of the LDC fork of LLVM 6: https://github.com/ldc-developers/llvm/tree/ldc-v6.0.1-3

The bootstrap compiler has been updated to v0.17.6 to support this: https://github.com/ldc-developers/ldc/releases/v0.17.6

[WebDrake]

I'm getting a massive list of errors from cmake when trying to build the LLVM part of this. Did something major change in the 5 => 6 build setup/requirements? (AFAICS it is strictly related to the LLVM 6 upgrade.)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-all" does not exist.
Call Stack (most recent call first):
  CMakeLists.txt:919 (add_lit_target)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-all" does not exist.
Call Stack (most recent call first):
  CMakeLists.txt:919 (add_lit_target)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-builtins" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/builtins/CMakeLists.txt:39 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-builtins" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/builtins/CMakeLists.txt:39 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-interception" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/interception/CMakeLists.txt:14 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-interception" does
  not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/interception/CMakeLists.txt:14 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-lsan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/lsan/CMakeLists.txt:36 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-lsan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/lsan/CMakeLists.txt:36 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-ubsan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/ubsan/CMakeLists.txt:71 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-ubsan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/ubsan/CMakeLists.txt:71 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-cfi" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/cfi/CMakeLists.txt:88 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-cfi" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/cfi/CMakeLists.txt:88 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-cfi-and-supported" does not
  exist.
Call Stack (most recent call first):
  projects/compiler-rt/test/cfi/CMakeLists.txt:92 (add_lit_target)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-cfi-and-supported"
  does not exist.
Call Stack (most recent call first):
  projects/compiler-rt/test/cfi/CMakeLists.txt:92 (add_lit_target)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-sanitizer" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/sanitizer_common/CMakeLists.txt:54 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-sanitizer" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/sanitizer_common/CMakeLists.txt:54 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-fuzzer" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/fuzzer/CMakeLists.txt:40 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-fuzzer" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/fuzzer/CMakeLists.txt:40 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-asan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:159 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-asan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:159 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-asan-dynamic" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:165 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-asan-dynamic" does
  not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:165 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-dfsan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/dfsan/CMakeLists.txt:28 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-dfsan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/dfsan/CMakeLists.txt:28 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-msan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/msan/CMakeLists.txt:54 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-msan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/msan/CMakeLists.txt:54 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-tsan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/tsan/CMakeLists.txt:100 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-tsan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/tsan/CMakeLists.txt:100 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-safestack" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/safestack/CMakeLists.txt:26 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-safestack" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/safestack/CMakeLists.txt:26 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-esan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/esan/CMakeLists.txt:29 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-esan" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/esan/CMakeLists.txt:29 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-scudo" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/scudo/CMakeLists.txt:38 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-scudo" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/scudo/CMakeLists.txt:38 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-ubsan-minimal" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/ubsan_minimal/CMakeLists.txt:23 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-ubsan-minimal" does
  not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/ubsan_minimal/CMakeLists.txt:23 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-profile" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/profile/CMakeLists.txt:26 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-profile" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/profile/CMakeLists.txt:26 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang" of target "check-xray" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/xray/CMakeLists.txt:39 (add_lit_testsuite)

CMake Error at cmake/modules/AddLLVM.cmake:1333 (add_dependencies):
  The dependency target "clang-headers" of target "check-xray" does not
  exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1354 (add_lit_target)
  projects/compiler-rt/test/xray/CMakeLists.txt:39 (add_lit_testsuite)

[WebDrake]

@thewilsonator can you please remove that approval? This patch is not yet good to go, for reasons described above :-\

[JohanEngelen]

this is probably because LDC's LLVM6 includes compiler-rt, which also needs clang sources to be present to build (at least on my mac) but those are unfortunately not part of LDC's LLVM repo. You could try with -DLLVM_BUILD_RUNTIME=OFF, although it'd be cool if the snap packages would ship the sanitizer binaries too (for which we'd have to add clang sources to the repo).

[WebDrake]

Makes sense, and is a sensible workaround for now, so I'll do that. But let's talk about how to handle clang sources. Assuming I can clone them from a repo, that should not be an issue.

[WebDrake]

Updated:

diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
index 330bc0b..5a73502 100644
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -93,6 +93,7 @@ parts:
     configflags:
     - -DCMAKE_BUILD_TYPE=Release
     - -DLLVM_BINUTILS_INCDIR=/usr/include
+    - -DLLVM_BUILD_RUNTIME=OFF
     - -DLLVM_TARGETS_TO_BUILD=X86\;AArch64\;ARM\;PowerPC\;NVPTX
     stage:
     - -*

[WebDrake]

I'll confirm back if this now works.

[JohanEngelen]

But let's talk about how to handle clang sources. Assuming I can clone them from a repo, that should not be an issue.

You'd have to clone them into the tools dir, the same way as lld is included.

Edit: But note that you should disable building clang. :)

[kinke]

You'll need -DCOMPILER_RT_INCLUDE_TESTS=OFF when including compiler-rt in the build, as (only) those require clang, see https://github.com/ldc-developers/llvm/blob/ldc-release_70/.circleci/config.yml#L58.

[WebDrake]

@kinke thanks, that seems to be working (... so far! build is ongoing ...).

Looking at the CircleCI config I note that there's also a discrepancy in LLVM targets (and experimental targets) that I ought to resolve, which I'll do in a follow-up PR.

On that note, @thewilsonator can you confirm which targets are required for dcompute's GPU support? The package already includes the NVPTX target, but what's the feasibility of adding a SPIR-V target as well? I don't see one listed in the LLVM 6 CMakeLists.txt target collection, but I do see an AMDGPU target -- would that be of use to you (and feasible for LDC to use)?

[WebDrake]

Updated following @kinke's suggestion:

diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
index 5a73502..165c1a0 100644
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -92,8 +92,8 @@ parts:
     plugin: cmake
     configflags:
     - -DCMAKE_BUILD_TYPE=Release
+    - -DCOMPILER_RT_INCLUDE_TESTS=OFF
     - -DLLVM_BINUTILS_INCDIR=/usr/include
-    - -DLLVM_BUILD_RUNTIME=OFF
     - -DLLVM_TARGETS_TO_BUILD=X86\;AArch64\;ARM\;PowerPC\;NVPTX
     stage:
     - -*

[WebDrake]

OK, this builds successfully; I'm taking it in.