Issues with variations in compile_commands.json and rsp.gcd files

[jgilardi]

With UnrealEngine 5.3.2 release (https://github.com/jgilardi/UnrealEngine/commit/072300df18a94f18077ca20a14224b5d99fee872) there are slight variations in the format of compile_commands.json when using UnrealBuildTool.exe mode=GenerateClangDatabase. This causes parsing issues with Unreal.nvim generate commands.

The issues are minor. Mostly string find skipping over the command args around here: https://github.com/zadirion/Unreal.nvim/blob/de4f81d4f0462796217db885d3ab2a11cc6e7542/lua/unreal/commands.lua#L485C19-L485C19

I've updated (hacked) the plugin to work with the latest compile_command.json syntax, however, I may have a few issues.

Is it normal for the .rsp to be transformed like this and lose all the extra flags after the includes?

RSP_BEFORE

"D:/devservoir/UnrealEngine/Engine/Source/Runtime/Experimental/Chaos/Private/Chaos/AABB.cpp"
/FI"D:/devservoir/UnrealEngine/Engine/Intermediate/Build/Win64/x64/UnrealEditor/Development/Chaos/Definitions.h"
/Fo"D:/devservoir/UnrealEngine/Engine/Intermediate/Build/Win64/x64/UnrealEditor/Development/Chaos/AABB.cpp.obj"
/sourceDependencies "D:/devservoir/UnrealEngine/Engine/Intermediate/Build/Win64/x64/UnrealEditor/Development/Chaos/AABB.cpp.dep.json"
@"..\Intermediate\Build\Win64\x64\UnrealEditor\Development\Chaos\Chaos.Shared.rsp"
/Zc:inline
/nologo
/Oi
/FC
/c
/Gw
/Gy
/utf-8
/wd4819
/DSAL_NO_ATTRIBUTE_DECLARATIONS=1
/permissive-
/Zc:strictStrings-
/Zc:__cplusplus
/D_CRT_STDIO_LEGACY_WIDE_SPECIFIERS=1
/D_SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS=1
/D_WINDLL
/D_DISABLE_EXTENDED_ALIGNED_STORAGE
/Ob2
/Ox
/Ot
/GF
/errorReport:prompt
/EHsc
/DPLATFORM_EXCEPTIONS_DISABLED=0
/Z7
/MD
/bigobj
/fp:fast
/Zo
/Zp8
/we4456
/we4458
/we4459
/we4668
/we4244
/we4838
/TP
/GR-
/W4
/std:c++20
/Zc:preprocessor
/wd5054

RSP_AFTER

-include "D:/devservoir/UnrealEngine/Engine/Intermediate/Build/Win64/x64/UnrealEditor/Development/Chaos/Definitions.h"
-include "D:/devservoir/UnrealEngine/Engine//Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h"
-include "D:/devservoir/UnrealEngine/Engine//Engine/Source/Runtime/Core/Public/Misc/EnumRange.h"
-std=c++20 -Wno-deprecated-enum-enum-conversion -Wno-deprecated-anon-enum-enum-conversion -ferror-limit=1 -Wno-inconsistent-missing-override

Request:

I was wondering if you could post an example compile_commands.json: both the initial .json and the modified one?
As well as some before/after of the .rsp files?

Thanks!

[xubury]

Hey, I have run into the same problem as yours. I also modified the rsp path finding a bit to fix some errors.

            i,j = line:find("%@\\\"")
            if i then
                local _,endpos = line:find("\\\"", j)
                local rsppath = line:sub(j+1, endpos-2)

The generation is sucessful, but clangd seems unable to find my project source files and some of the engine headers. I wonder if you have any progress solving this?

[xubury]

Hello, after some investigations I believe the issue is caused by UBT using a MSVC 2022 new feature(sourceDependencies) that is clang incompatible. See: https://learn.microsoft.com/en-us/cpp/build/reference/sourcedependencies?view=msvc-170

I am not sure what's the original idea of @zadirion to replace cl with clang++. But I have successfully get it working by retaining the orignal content of the rsp file and compile_commands.json, in addition of some extra headers. I made my own fork if you want to check it out: https://github.com/xubury/Unreal.nvim

[zadirion]

@xubury Heya, thanks for looking into this issue. The reason why I replaced cl with clang++ is because microsoft's cl doesn't supply AST info to the clang server (as far as I could tell), which means vim's built-in LSP won't have any AST info. Maybe I'm wrong? Are you sure cl.exe is working with vim's LSP?

[xubury]

@xubury Heya, thanks for looking into this issue. The reason why I replaced cl with clang++ is because microsoft's cl doesn't supply AST info to the clang server (as far as I could tell), which means vim's built-in LSP won't have any AST info. Maybe I'm wrong? Are you sure cl.exe is working with vim's LSP?

Yes, I think clangd can work with cl. I have used clangd with gcc, clang and msvc across different projects. I think compiler itself doesn't matter, I remember clangd can work with CUDA compiler too. In my understanding, AST is analyzed by clangd server as long as you have the correct compile_command.json and c++ files. Though I have to add Engine/Source/Runtime/Engine/Public/Engine.h to extraIncludes to prevent clangd complaining for missing GEngine.

[zadirion]

@xubury it's the compiler itself that actively sends ast information to clang server. That's why i doubt microsoft implents that in their compiler, last time i checked it didnt

[xubury]

@xubury it's the compiler itself that actively sends ast information to clang server. That's why i doubt microsoft implents that in their compiler, last time i checked it didnt

I don't think so. I recall in very early stage clangd's codebase is part of the clang compiler. But it's not the case now. Here's my log from nvim's built-in lsp. lsp.log From the log there's clearly a AST worker from clangd that's sending info to neovim.

"C:\\Users\\bury\\AppData\\Local\\nvim-data\\mason\\bin\\clangd.CMD"    "stderr"    'I[20:37:01.846] ASTWorker building file C:\\Users\\bury\\Documents\\Unreal Projects\\Infinite\\Source\\Infinite\\Private\\PortalVolume.cpp version 0 with command \r\n[C:\\UE_5.3\\Engine\\Source]\r\n"C:\\\\Program Files\\\\Microsoft Visual Studio\\\\2022\\\\Community\\\\VC\\\\Tools\\\\MSVC\\\\14.38.33130\\\\bin\\\\Hostx64\\\\x64\\\\cl.exe" --driver-mode=cl

[xubury]

I think the compiler specified in compile_commands.json is just to tell which compiler backend you are using when building the code. The AST is provided by clangd which is a part of the front end of clang compiler.

[xubury]

After some googling. It appears that the compiler parameter in compile_commands.json tell what "version" of compiler arguments you are using. Clang will have cl arugments transfer to llvm compatible arugments. Maybe some older version it doesn't?

// llvm_root\tools\clang\tools\driver\driver.cpp
const DriverSuffix *FindDriverSuffix(StringRef ProgName, size_t &Pos) {
  // A list of known driver suffixes. Suffixes are compared against the
  // program name in order. If there is a match, the frontend type is updated as
  // necessary by applying the ModeFlag.
  static const DriverSuffix DriverSuffixes[] = {
      {"clang", nullptr},
      {"clang++", "--driver-mode=g++"},
      {"clang-c++", "--driver-mode=g++"},
      {"clang-cc", nullptr},
      {"clang-cpp", "--driver-mode=cpp"},
      {"clang-g++", "--driver-mode=g++"},
      {"clang-gcc", nullptr},
      {"clang-cl", "--driver-mode=cl"},
      {"cc", nullptr},
      {"cpp", "--driver-mode=cpp"},
      {"cl", "--driver-mode=cl"},
      {"++", "--driver-mode=g++"},
  };

[jgilardi]

Sorry for the necro, but I didn't login to github for a few months.

With @xubury 's suggestions I was able to get things to work. Notably including Engine.h and a fix similar to the gsub one mentioned above.

Turns out, most issues I ran into were because of a quirk of my folder structure: I don't keep my UE project folders inside the UnrealEngine directory.

My setup is like this:

devservoir/
   UnrealEngine
   ueproj
      Project01
      Project02
      ...

Some issues were intrinsic to clangd. It has issues swapping between Project and UnrealEngine directories when they're setup as shown. For example I could use the lsp to navigate Engine files if I started from the UnrealEngine directory, or I could navigate Project files if I started from a project directory. However, I could not navigate TO the engine files from the Project directory.

I was able to get things working though. The trick is absolute paths:

From the project directory, for the plugin, generate a UnrealNvim.json. Make the engine directory an absolute path.

"EngineDir": "D:\\devservoir\\UnrealEngine",   

Convert all paths in the compile_commands.json to absolute ones. FROM

{
"file": "D:\\devservoir\\UnrealEngine\\Engine\\Source\\ThirdParty\\libSampleRate\\Private\\src_zoh.cpp",
"command": "\"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\Hostx64\\x64\\cl.exe\" @\"..\\Intermediate\\Build\\Win64\\x64\\UnrealEditor\\Development\\UELibSampleRate\\src_zoh.cpp.obj.rsp.gcd\"",
"directory": "D:\\devservoir\\UnrealEngine\\Engine\\Source"
},

TO

"file": "D:\\devservoir\\UnrealEngine\\Engine\\Source\\ThirdParty\\libSampleRate\\Private\\src_zoh.cpp",
"command": "\"C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.36.32532\\bin\\Hostx64\\x64\\cl.exe\" @\"D:\\devservoir\\UnrealEngine\\Engine\\Intermediate\\Build\\Win64\\x64\\UnrealEditor\\Development\\UELibSampleRate\\src_zoh.cpp.obj.rsp.gcd\"",
"directory": "D:\\devservoir\\UnrealEngine\\Engine\\Source"
},

There were a few minor issues to fix in Unreal.nvim to parse compile_commands.json with absolute paths. Just make sure that it can read and write the rsp files it wants.

Finally, update the additional includes as described above to fix some clangd warnings. Ex:

function ExtractRSP(rsppath)
    local extraIncludes = {
        "Engine/Source/Runtime/CoreUObject/Public/UObject/ObjectMacros.h",
        "Engine/Source/Runtime/Core/Public/Misc/EnumRange.h",
        "Engine/Source/Runtime/Engine/Public/EngineMinimal.h",
        -- see issues: https://github.com/zadirion/Unreal.nvim/issues/4
        "Engine/Source/Runtime/Engine/Public/Engine.h",
    }