Build failed with: "llvm-build: error: invalid native target: 'SystemZ' (not in project)"

[QianNangong]

Steps to reproduce

git clone https://github.com/avast-tl/retdec
cd retdec
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7

Output

CMake Error at CMakeLists.txt:539 (message):
  Unexpected failure executing llvm-build: Usage: llvm-build [options]

  llvm-build: error: invalid native target: 'SystemZ' (not in project)

-- Configuring incomplete, errors occurred!

Environment

SUSE Linux Enterprise Server 12 SP3 (s390x)

[s3rvac]

Thank you for the report. This may be related to #2.

[PeterMatula]

LOL, do I understand it correctly that you are trying to build RetDec on an IBM mainframe? We have never really tried to build it on anything other than x86, but ok, we can give it a shot.

1. I fixed #2, but I don't think this will help you.

2. Lets cut the middleman - do not build entire RetDec, try to build LLVM repo itself:

   git clone https://github.com/avast-tl/llvm.git
   cd llvm
   mkdir build && cd build
   cmake ..
   make

3. This should fail with the same message you got before in the cmake .. step.

4. Now, there are 2 possibilities:

   1. LLVM 3.9.1 cannot be built on you system. There is probably nothing we can do.
   2. Vanilla LLVM 3.9.1 can be built on your system, but our aggressive removal of sources and targets in the faster-build-2 branch (merged to master) screwed something up. Hopefully we can fix this.

5. Find out which one it is - get to the state before the faster-build-2 merge and try to build it. If it passes, we removed something that we should not. If it fails, it never worked.

   git clone https://github.com/avast-tl/llvm.git
   cd llvm
   git checkout e7431aa
   mkdir build && cd build
   cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=NO -DLLVM_REQUIRES_RTTI=YES -DLLVM_ENABLE_TERMINFO=YES -DLLVM_BUILD_TOOLS=OFF -DLLVM_BUILD_UTILS=OFF -DLLVM_BUILD_RUNTIME=OFF -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7 -DPYTHON_EXECUTABLE=/usr/bin/python3
   make

   I don't know if setting PYTHON_EXECUTABLE is really required on you system, but set it to the absolute path to your python3 just to be sure.

6. Let us know what happened and we will take it from there.

[QianNangong]

Thanks! I'll have a try.

[PeterMatula]

If step 5. fails, you can also try it with the initial commit (it has slightly different directory and CMake structure compared to the current state - no need for all those defines in cmake ..):

git clone https://github.com/avast-tl/llvm.git
cd llvm
git checkout 4876ee1
mkdir build && cd build
cmake ..
make

[QianNangong]

Step 5 does work, I can build a working LLVM on the machine.

[PeterMatula]

Ok, I will try to inspect the changes I made and identify possible causes. However, I'm not able to reproduce this, so it won't be easy.

[QianNangong]

It occurs only when commit fdabb615a3c221aa31205064d13ad21583e1540d is applied. Any commits older than 18fa1af4afc0c3cbb296b8307df20ad1dc795279 will work.

[PeterMatula]

Ok, thanks.

[PeterMatula]

Try this for the current master:

git clone https://github.com/avast-tl/llvm.git
cd llvm
mkdir build && cd build
cmake .. -DCMAKE_C_COMPILER=gcc-7 -DCMAKE_CXX_COMPILER=g++-7 -DLLVM_TARGET_ARCH=X86
make

[QianNangong]

It works, thank you! Should I close this issue?

[PeterMatula]

No, this was just a test that we found the problem. I will make this change directly in LLVM's CMake and update the reference to LLVM in the main RetDec repository. Then, you won't have to define it every time you build. I will close the issue after.

Thank you for helping us identify this problem. Like i said, we have never tried it on other architectures, so I'm not sure it will work and you won't encounter some other problems.

[QianNangong]

Thanks for your hard work.

[PeterMatula]

Is your system (s390x) big endian? My colleague says he would be very surprised if everything worked ok :-D If you encounter other problems, please report them. Also it would be amazing if you were able to run regression tests and let us know if something fails - upload the log to a new issue. Then we would at least know we have a problem.

[PeterMatula]

Summary of what happened here:

• We don't really need LLVM's machine targets for RetDec. We are not lowering LLVM IR to target assembly.
• When speeding up the LLVM build (in order to run RetDec build in public CI services), we removed all the targets but x86 from libs/Target. We don't actually need the x86 but it is too hard to hack LLVM to build without any targets.
• We force this target with -DLLVM_TARGETS_TO_BUILD=X86.
• However, LLVM build system is adding the native target to targets from LLVM_TARGETS_TO_BUILD.
• On x86 systems, the native target is x86 and everything is OK.
• In this case, the native target was s390x that got translated to SystemZ, and cmake .. failed because module for this target was removed.
• We force x86 to LLVM_TARGET_ARCH. This variable is used to generate LLVM_NATIVE_ARCH. Now, x86 module is used on all systems.

[QianNangong]

Yes, the system is big endian. I'm building the RetDec Regression Tests Framework to test if it really works.