"unable to handle compilation, expected exactly one compiler job" - static compilation from multiple source files

[jmrgibson]

I'm trying to compile this for an embedded target using a very non-standard compiler.

I've already removed all the unsupported compiler flags from the compile_commands.json, and pasting the command from compile_commands.json into clang works (gives me some expected errors for missing typedefs). Sourcetrail however seems to be stuck on translation units coming from multiple .c source units.

I can't post the full error due to privacy reasons, but here is a sample (can probably a more detailed one if needed)

Is source trail supposed to be able to support static object file compilation targets from multiple source files?

unable to handle compilation, expected exactly one compiler job in '' "clang-tool" "-cc1" "-triple" "x86_64-apple-macosx10.14.0" "-Wdeprecated-objc-isa-usage" "-Werror=deprecated-objc-isa-usage" "-fsyntax-only"... 
compiler flags...
includes...
"some_source_file.c";  "clang-tool" "-cc1" "-triple" "x86_64-apple-macosx10.14.0" "-Wdeprecated-objc-isa-usage" "-Werror=deprecated-objc-isa-usage" "-fsyntax-only"... 
compiler flags...
includes...
"some_source_file2.c";   (this error is repeated  for every input source file for the translation unit)

[mlangkabel]

Hi @jmrgibson, thanks for reaching out to us.

The compile command you posted already looks like it is meant to be two separate commands. If you would split it at the ";", each of the two parts would provide a single source file, specify "clang-tool" to be run and provide all the necessary flags. Can you test that approach?

[jmrgibson]

Hi @mlangkabel , I tried splitting the entries in "compile_commands.json" into multiple entries (one for each source file), and that worked. However I would have expected sourcetrail to handle this by itself, since multi-file compilation is a standard part of clang?

My original compile_commands.json

[
    {
        "command": "/some_custom_compiler -o components/TorquePath.obj -std=c99 -g -s -os -o3 -DLOCAL_BUILD=1 -I/some_custom_compiler/include components/TorquePath.c components/Utils.c components/math.c",
        "file": "components/TorquePath.c",
        "directory": "/Users/jgibson/build"
    }
]

Modified compile_commands.json

[
    {
        "command": "/some_custom_compiler -o components/Utils.obj -std=c99 -g -s -os -o3 -DLOCAL_BUILD=1 -I/some_custom_compiler/include components/Utils.c",
        "file": "components/Utils.c",
        "directory": "/Users/jgibson/build"
    },
    {
        "command": "/some_custom_compiler -o components/TorquePath.obj -std=c99 -g -s -os -o3 -DLOCAL_BUILD=1 -I/some_custom_compiler/include components/TorquePath.c",
        "file": "components/TorquePath.c",
        "directory": "/Users/jgibson/build"
    },
    {
        "command": "/some_custom_compiler -o components/math.obj -std=c99 -g -s -os -o3 -DLOCAL_BUILD=1 -I/some_custom_compiler/include components/math.c",
        "file": "components/math.c",
        "directory": "/Users/jgibson/build"
    }
]

[mlangkabel]

Thanks for testing and reporting the results! I personally haven't heard of Clang being able to handle multi-file compile commands but I would be happy to investigate! What would you expect Clang to do, if you provide the "original compile_commands.json"? Compile the 3 .c files to 3 different .o files? Do you know if there is a Clang documentation that describes this kind of compilation unit?

[jmrgibson]

// foo1.c
#include "foo2.h"
int main(void){
   bar();
}

// foo2.h
void bar(void);

// foo2.c
#include "foo2.h"
#include "stdio.h"
void bar(void){
    printf("hi");
}

then compile by running

clang -o foo.o -I<path_too_foo2.h> foo1.c foo2.c

will produce a single object file foo.o that contains object info from both foo1.c and foo2.c.

objdump -t foo.o

foo.o:  file format Mach-O 64-bit x86-64

SYMBOL TABLE:
0000000100000000 g     F __TEXT,__text  __mh_execute_header
0000000100000f70 g     F __TEXT,__text  _bar
0000000100000f60 g     F __TEXT,__text  _main
0000000000000000         *UND*  _printf
0000000000000000         *UND*  dyld_stub_binder

Lots of embedded & other high performance systems work this way when you are optimizing for speed or code size over compilation speed

[mlangkabel]

Thanks for providing the example. So the clang compiler allows for multiple source files to be compiled in one go. But it looks like the clang compilation database format doesn't really support this yet: https://clang.llvm.org/docs/JSONCompilationDatabase.html

Can you tell us how you generated the compilation database that you used for Sourcetrail?

[jmrgibson]

I've been using this script, https://github.com/mongodb/mongo/blob/master/site_scons/site_tools/compilation_db.py, but modified to a) remove all of our compiler specific defines b) split multi-file compilation into single lines

Ideally sorucetrail would handle multi-source inputs and I could remove step b)

[mlangkabel]

Nice to know of this possibility to generate a compile_commands file.

So, for Sourcetrail internally we do not modify the provided compile commands a lot. We pass them to the Clang tool almost exactly as we got them. And as I have suspected before, it appears that Clang would not understand these multi-file compilations: In Tooling.cpp getCC1Arguments() an explicit error is reported if the compilation does not contain exactly one job.

Of course we could step in and split the compile command the way you suggested. But then again, this may introduce errors or unexpected behavior for other users: Assume that some user has some flag called -VerySpecialInclude ./foo.h. In this case Clang doesn't know the -VerySpecialInclude flag and discards it. What stays in the pipeline is ./foo.h. If we would now split this compile command, we would add a translation unit for ./foo.h, which is definitely not what that user wants. We discussed this with the team and came to the conclusion that we don't want to deviate from how the compile commands are processed by Clang. Sorry.

So it would be either to fix the mongo tools code to generate compatible compile commands (I just checked their jira trackers and didn't find a place to report those tools related issues. If you happen to know where to report this issue, please feel free to reference this GitHub issue). Or report this issue to the Clang developers. But it would surely take a while until this feature is implemented, released and used by Sourcetrail.

[jmrgibson]

Not much point changing the mongodb script, we'll just keep our internal one for now. Strange that clang supports multi-file compilation, but the clang-tool doesn't support the resulting compile-commands. I'll file a clang-tools bug.

[mlangkabel]

Awesome, thanks!