Clang compiler stuck and lots of memory are used when combining -O2 with -g

[vcarreira]

I have a table with a considerable number of columns. When compiling in debug (-O0) or in release (-O3) everything works fine.

However, to be able to generate a dSYM file, I have to turn on debugging symbols (-g). Any attempt to combine -O3 (or any other kind of optimization) with -g makes the compiler consume lots of memory and never finish.

Clang version:

Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I have attached an example that hangs the compiler. Example_hpp.txt Example_cpp.txt

Compiling the above example with the following flags will never finish and clang will consume up to 5GB of memory: clang++ -std=c++17 -stdlib=libc++ -O2 -g -c Example.cpp

Interesting enough, without changing the number of columns, by just replacing fields 1 to 3 with optionals, the compiler can make progress.

    std::optional<std::int64_t> field1;
    std::optional<std::int64_t> field2;
    std::optional<std::int64_t> field3;

Any help would be appreciated

[fnc12]

what RAM has your machine and what OS?

[vcarreira]

what RAM has your machine and what OS?

MacBook Pro I7 16GB of RAM (macOS Catalina). Tried both Apple Clang compiler (Xcode 12.4) and Clang 11.1.0 installed via homebrew

[fnc12]

let me check it tomorrow

[vcarreira]

let me check it tomorrow

Any update regarding this issue? The same behavior can be observed on BigSur/Catalina, MacBookPro/Mac mini, either with Apple Clang 12 or Clang 11.1.0 installed with homebrew

[fnc12]

@vcarreira sorry I was busy with another issue. I'll check it today

[fnc12]

I've started compilation and looks like my Xcode is stuck also. I leave my mac working and will be back in the evening (now it is morning) in case it needs more time. Will see

[fnc12]

So I am back and Xcode is still compiling. I can reproduce your issue. BTW if you comment out the latest line:

return storage.get_all<Entity>(where(c(&Entity::dataset_id) == dataset_id));

and replace it with

return {};

then everything will compile well.

[vcarreira]

@fnc12 Yep but that's not the point :) That line triggers the template instantiation for a query. Without that line we just have the storage definition.

It will also compile instantaneously if you modify the columns and replace the field1 to field3 with:

std::optional<std::int64_t> field1;
std::optional<std::int64_t> field2;
std::optional<std::int64_t> field3;

So, my question is what can be done to compile with -O2 and -g without making the compiler go into an endless loop?

[fnc12]

@fnc12 Yep but that's not the point :) That line triggers the template instantiation for a query. Without that line we just have the storage definition.

Yeah you're right. I understand. It also works if you reduce the amount of columns by 5 or 6. This looks like a clang issue but I'd like to find a clue. Perfect solution is debugging clang but it is not what I am going to do sorry. I think we can find a quicker and easier way to resolve this. You can move this function call into a different cpp file and compile it separately using -O3 or -O1 and link it against your binary as a static library. Or it is not an option?

[vcarreira]

@fnc12 It’s always an option, although we are going to be kind of “blind” if a crash happens with something related with the database.

The code I posted was just a tiny test to demonstrate the issue. The real code is much more complex and I was hoping for a workaround that would not involve isolating the ORM module. But I understand that you don’t have time to dig into clang deep debug.

Anyway, thanks for awesome work with sqlite_orm

[fnc12]

@vcarreira ok let's find a clue. Just give me some time

[fnc12]

@vcarreira I cannot find any clue. Looks like we need to debug compiler. Can you make an issue in clang repo? I guess guys there can easily say what is wrong.

[fnc12]

@vcarreira are you here?

[fnc12]

@vcarreira are you here?

[vcarreira]

@fnc12 Sorry... being busy and missed the email notification.

I've just tested with clang-12 and it works. It was a nasty compiler bug on clang.

Thank you

[fnc12]

@vcarreira oh it is nice. I am glad. Please feel free to ask anything here

[czln]

arm-aarch-gcc version 12 came up with the same problem in old version of sqlite_orm, but not with the newest (commit 4861e407eb94740e12ea7fd496945ed5702c3f4c).

Is this truely a compiler problem?

was the problem already be located and fixed?

[trueqbit]

@czln I wouldn't necessarily say that we have found the problem explicitly. However, for v1.8 I have streamlined the entire template code machinery. The tuplification of the 'storage' implementation and the improved tuple filtering and iteration seem to have a positive effect on the resources required by the compiler.

[czln]

@trueqbit OK, so it's kind of fixed thx

[trueqbit]

@czln Absolutely, and thx for reporting your success. While you're at it, it would be useful if you could analyze the builds so we can get an idea. Clang has a -ftime-trace option, and the results can be visualized with the chrome://tracing tool or the ClangBuildAnalyzer.