Open giulianobelinassi opened 6 months ago
Ping
ping 2
ping 3
@EugeneZelenko do you have any code pointers that we could check and try to solve this issue ourselves? Thanks in advance!
This have to do with PCH generation. The following code creates a Preprocessor to compile them:
if (PrecompilePreambleAfterNParses > 0) {
PreambleRebuildCountdown = PrecompilePreambleAfterNParses;
OverrideMainBuffer =
getMainBufferWithPrecompiledPreamble(PCHContainerOps, *Invocation, VFS);
getDiagnostics().Reset();
ProcessWarningOptions(getDiagnostics(), Invocation->getDiagnosticOpts());
}
from ASTUnit::LoadFromCompilerInvocation
I encountered this problem when I was browsing fuzzel codebase with clangd. I hope this gets fixed.
as mentioned by @giulianobelinassi, this is due to usage of PCHs by ASTUnit or clangd for quickly reparsing sources.
You can imagine this as running two separate compilations, one for all the includes (preamble) in your file, and another using this preamble for includes and parsing rest of the source file. As a result both compilations initialize __COUNTER__
to 0
.
The same also likely to affect all sorts of modular builds, as each module will be build with its own __COUNTER__
and merged together in a not-so-strictly ordered fashion. cc @dmpolukhin.
In the case of tooling we at least have a single preamble and single main file parse, so we can work around by storing the final count in PCHs and re-using that during main file parse. But this solution won't work when there are multiple PCHs involved, and it gets even nastier for actual modules.
https://isocpp.org/files/papers/P3384R0.html#design-considerations is recognizing this difficulties but isn't really proposing a solution. I think we'd need someone to put a lot of thought into either making it work in general or find a way to restrict them for a modules-friendly world. cc @AaronBallman
For anyone interested in remedying tooling adjacent symptoms by storing/restoring __COUNTER__
in PCHs, http://go/llvm-project/clang/include/clang/Frontend/PrecompiledPreamble.h is the abstraction used by both clangd and ASTUnit for managing preamble-PCHs.
From our observations and from the code fragment in the initial post. It seems that people usually use __COUNTER__
as just a unique value, they don't need monotonically increasing values and any unique value would work but it has to be reproducible always produce the same AST from the same inputs. So I was thinking about some seed from filename (perhaps with some path parts, for example like it was specified in command line). __COUNTER__
value will be next random value generated based on this seed. Every time when module is loaded, the seed of the module and the main TU seed are combined to produce new seed value.
For our usage we decided to disable PCH altogether, since it avoids this problem and we noticed significant speedups in clang-extract. This can be done changing:
auto AST = ASTUnit::LoadFromCompilerInvocation(
CInvok, PCHContainerOps, Diags, FileMgr, false, CaptureDiagsKind::None, 1,
TU_Complete, false, false, false);
to
auto AST = ASTUnit::LoadFromCompilerInvocation(
CInvok, PCHContainerOps, Diags, FileMgr, false, CaptureDiagsKind::None, 0,
TU_Complete, false, false, false);
With the following files:
header.h:
test.c:
compiling with:
Results in the input being accepted.
Now with libtooling, the following reproducer (
$ clang++ -g llvm-repro.cpp -lclang-cpp -lLLVM
):Results in the output always being rejected with the following output:
Notice how it is exactly the same code, but one is being compiled through clang, whereas the other is being compiled through libtooling.
I'd say this is undesired behavior and a bug.