Unreal Engine 4 support

[Charusso]

• What you were trying to do. make Unreal Engine 4 from source to create compile_commands.json.

• The version number of CodeChecker c0664e3 (6.9.1)

What behaviour was expected instead of what happened.

1. As usual I tried the following to produce the database:

   • CodeChecker log -b "make -j13" -o compile_commands.json

   • Outcome: Build failed.

2.1. I have followed the advice by Timothee "TTimo" Besset with Bear:

• He has a blog post about his research working with Sourcetrail:
  • https://ttimo.typepad.com/blog/2018/09/sourcetrail-with-unreal-engine-4-a-quick-primer.html
• This is in a nutshell:

git clone --single-branch --branch ttimo https://github.com/TTimo/Bear.git
cd Bear/
cmake .
make all
sudo make install
cd ..
bear ./Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch

• Outcome: it worked well, created a 76,1 MB compile_commands.json.
• It has lot of tricks to achieve that (made by him):
  • https://github.com/rizsotto/Bear/pull/224
  • https://github.com/rizsotto/Bear/issues/194
  • https://github.com/rizsotto/Bear/issues/173

2.2. I wanted to use that Bear-made compile_commands.json:

• The usual command: CodeChecker analyze compile_commands.json -o ...

• Outcome: Other software like Sourcetrail works perfectly with that, but CodeChecker is not:

  clang-9: error: unknown argument: '-main-file-name'
  clang-9: error: unknown argument: '-mrelocation-model'
  clang-9: error: unknown argument: '-mconstructor-aliases'
  clang-9: error: unknown argument '-munwind-tables', did you mean '-funwind-tables'?
  clang-9: error: unknown argument: '-fdeprecated-macro'
  clang-9: error: unknown argument: '-fdebug-compilation-dir'
  clang-9: error: unknown argument '-ferror-limit', did you mean '-ferror-limit='?
  clang-9: error: unknown argument '-fmessage-length', did you mean '-fmessage-length='?
  clang-9: error: invalid thread model '-masm-verbose' in '-mthread-model -masm-verbose' for this target
  ...

---

3. Copied the tricky build command from TTimo to CodeChecker:

   • CodeChecker log -b "./Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch" -o compile_commands.json

   • Outcome: successfully built, but the compile_commands.json was empty, so failed.

   • I have hard-coded the arguments into Build.sh and the outcome was the same so the parsing is cool I think.

4. Tried out the hack made by TTimo combined with his build command with CodeChecker:

   • export CC_LOGGER_GCC_LIKE="gcc:g++:clang:cc1"
   • Note: cc1plus may would be involved so I tried "gcc:g++:clang:cc1:cc1plus" as well.
   • CodeChecker log -b "./Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch" -o compile_commands.json

• Outcome: same as 3.: successfully built, but the compile_commands.json was empty, so failed.

---

Additional information

5. It is too heavy
   • If you try to produce a compile_commands.json with ninja it immediately run into a trouble:
   • Argument list too long.

• There is two workaround to split the overhead:

  • By a random Google search:
  • https://stackoverflow.com/questions/43184251/cmake-command-line-too-long-windows
  • By the official issue of that problem ( https://github.com/ninja-build/ninja/issues/1261 ):
    • https://ninja-build.org/manual.html#ref_rule

• If you check out the CMakeLists.txt is has the following (mentioned in the first link):

  set(CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS 1 CACHE BOOL "" FORCE)
  set(CMAKE_CXX_USE_RESPONSE_FILE_FOR_INCLUDES 1 CACHE BOOL "" FORCE)

• The UE4 is that heavy it has that workaround here by default.

• Note: ninja cannot handle it as well: If you try to build it hacking CMakeLists.txt with the above trick: set(CMAKE_NINJA_FORCE_RESPONSE_FILE 1 CACHE INTERNAL ""), it tries to compile 8412 compilation unit, but after a while there is too much error and it give up.

6. How to reproduce the environment

   • Follow the official tutorial:
   • https://docs.unrealengine.com/en-us/Platforms/Linux/BeginnerLinuxDeveloper/SettingUpAnUnrealWorkflow

   • Before make create a copy of the UnrealEngine folder because there is no make clear.

7. The environment

   • The project folder without built stuff is 69,3 GB.
   • After build and ./Engine/Binaries/Linux/UE4Editor to check if it is working the folder is 78 GB.

   • Building is about 5120 compilation unit with the tricky command.

8. The tricky command explained /Engine/Build/BatchFiles/Linux/Build.sh UE4Game Linux Development -disableunity -nopch

   • disableunity: process every compilation unit separately (5th post by gmpreussner):
   • https://forums.unrealengine.com/development-discussion/engine-source-github/121146-how-to-compile-in-non-unity-mode?p=1346959#post1346959
   • nopch: no pre-compiled header (2nd post by ExtraLifeMatt):
   • https://forums.unrealengine.com/development-discussion/c-gameplay-programming/1552216-what-is-pch-and-badaptiveunitydisablespch

---

Summary

I cannot decide if we would

• 2.) use hacked Bear-made compile_commands.json like Sourcetrail does,
• 4.) support cc1/(cc1plus?) hooks better,
• 5.) support CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS/INCLUDES better.

The best would be just a simple CodeChecker log -b "make -j13" -o compile_commands.json.

[csordasmarton]

@Charusso I think #2092 solved your problem. Could you please try it out?

[axsaucedo]

@Charusso since UE2.24 you can actually use the UnrealBuildTool to generate the compile_commands.json. You just need to run "<path_to_ue4.24>/Engine/Binaries/DotNet/UnrealBuildTool.exe -mode=GenerateClangDatabase -project".uproject" Development Win64". This was tested in Windows + WSL. The output is a compile_commands.json file in your Unreal Engine base path.

Currently I am using it for the CCLS languag server for code checking in editor, however it doesn't seem to work fully as still several errors appear, but it may be enough for your usecase.

Here is an ongoing thread in the forums where it's being discussed as well, and includes further references to the documentation that mentions it in release notes https://forums.unrealengine.com/development-discussion/c-gameplay-programming/1525128-linux-rdm-compile_commands-json

[blasco]

@axsaucedo Is this working in Linux? I'm trying to do the same but when I run:

 /home/user/Programs/UnrealEngine/Engine/Build/BatchFiles/Linux/Build.sh/Build.sh -project=/home/user/Documents/Unreal/MyProject/MyProject.uproject -mode=GenerateClangDatabase UE4Game Linux Development

I get an error complaining that clang is not installed, but it is:

 clang --version

 clang version 10.0.0-4ubuntu1 
 Target: x86_64-pc-linux-gnu
 Thread model: posix
 InstalledDir: /usr/bin

[gyorb]

There is a comment in the mentioned forum discussion about the same problem, that the Build.sh script can not generate a compile command database file right now on linux.

10-11-2020, 01:47 PM Same situation here, not able to use -mode=GenerateClangDatabase it complaints about not finding clang, which is insatlled.