[analyzer] Plist macro-expansion crash

[steakhal]

Bugzilla Link	48358
Version	11.0
OS	Linux
Attachments	complete stack trace, add lit test as reproducer
CC	@devincoughlin

Extended Description

Here is the minimal repro (attached patch file):

// RUN: %clang_analyze_cc1 -analyzer-checker=core %s  \
// RUN:   -analyzer-output=plist -o %t.plist \
// RUN:   -analyzer-config expand-macros=true

#define STRANGE_FN(x) STRANGE_FN(x, 0)
void test_strange_macro_expansion() {
  char *path;
  STRANGE_FN(path); // no-crash
  // expected-warning@-1 {{implicit declaration of function}}
  // expected-warning@-2 {{1st function call argument is an uninitialized value}}
}

// CHECK: <key>name</key><string>STRANGE_FN</string>
// CHECK-NEXT: <key>expansion</key><string>STRANGE_FN(path, 0)</string>

Analyzer call (debug build):

./bin/clang -cc1 -internal-isystem git/llvm-project/build/debug/lib/clang/12.0.0/include -nostdsysteminc -analyze -analyzer-constraints=range -setup-static-analyzer -analyzer-checker=core git/llvm-project/clang/test/Analysis/plist-macros-with-expansion.c     -analyzer-output=plist -o tmp.plist    -analyzer-config expand-macros=true

Assertion triggered:

clang: ../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:1261: {anonymous}::MacroExpansionInfo getMacroExpansionInfo(const {anonymous}::MacroParamMap&, clang::SourceLocation, const clang::Preprocessor&): Assertion `TheTok.is(tok::r_paren) && "Expanded macro argument acquisition failed! After the end of the loop" " this token should be ')'!"' failed.

Relevant part of the backtrace:

 #​9 0x00007efe6fb5cf77 getMacroExpansionInfo((anonymous namespace)::MacroParamMap const&, clang::SourceLocation, clang::Preprocessor const&) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:1259:0
#&#8203;10 0x00007efe6fb5c2ba getMacroNameAndPrintExpansion((anonymous namespace)::TokenPrinter&, clang::SourceLocation, clang::Preprocessor const&, (anonymous namespace)::MacroParamMap const&, llvm::SmallPtrSet<clang::IdentifierInfo*, 8u>&) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:1015:0
#&#8203;11 0x00007efe6fb5c4d9 getMacroNameAndPrintExpansion((anonymous namespace)::TokenPrinter&, clang::SourceLocation, clang::Preprocessor const&, (anonymous namespace)::MacroParamMap const&, llvm::SmallPtrSet<clang::IdentifierInfo*, 8u>&) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:1050:0
#&#8203;12 0x00007efe6fb5c108 getExpandedMacro(clang::SourceLocation, clang::Preprocessor const&, clang::cross_tu::CrossTranslationUnitContext const&) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:1002:0
#&#8203;13 0x00007efe6fb59900 (anonymous namespace)::PlistPrinter::ReportMacroExpansions(llvm::raw_ostream&, unsigned int) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:393:0
#&#8203;14 0x00007efe6fb5a783 (anonymous namespace)::PlistDiagnostics::printBugPath(llvm::raw_ostream&, llvm::DenseMap<clang::FileID, unsigned int, llvm::DenseMapInfo<clang::FileID>, llvm::detail::DenseMapPair<clang::FileID, unsigned int> > const&, clang::ento::PathPieces const&) git/llvm-project/build/debug/../../clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp:603:0

Full trace attached.

Besides this, I've observed that the handwritten macro expansion in plist diagnostics is non-conformant. There are several examples where it produces the wrong expansion. Such as for this:

• for the generated plist: https://godbolt.org/z/YhE8zY
• for the expected macro expansions: https://cppinsights.io/lnk?code=dm9pZCBjbGFuZ19hbmFseXplcl93YXJuSWZSZWFjaGVkKCk7CgoKI2RlZmluZSByZXRBcmcoeCkgeAojZGVmaW5lIHJldEFyZ1VuY2xvc2VkIHJldEFyZyhjbGFuZ19hbmFseXplcl93YXJuSWZSZWFjaGVkKCkKCgojZGVmaW5lIEJCIENDCiNkZWZpbmUgYXBwbHlJbnQgQkIoaW50KQojZGVmaW5lIENDKHgpIHJldEFyZ1VuY2xvc2VkCgoKdm9pZCB1bmJhbGFuY2VkTWFjcm9zKCkgewogIGFwcGx5SW50ICk7Cn0KCiNkZWZpbmUgZXhwYW5kQXJnVW5jbG9zZWRDb21tYUV4cHIoeCkgKHgsIGNsYW5nX2FuYWx5emVyX3dhcm5JZlJlYWNoZWQoKSwgMQojZGVmaW5lIGYgZXhwYW5kQXJnVW5jbG9zZWRDb21tYUV4cHIKCnZvaWQgdW5iYWxhbmNlZE1hY3JvczIoKSB7CiAgaW50IHggPSAgZihmKDEpKSAgKSk7Cn0=&std=cpp2a&rev=1.0

The code:

void clang_analyzer_warnIfReached();

#define retArg(x) x
#define retArgUnclosed retArg(clang_analyzer_warnIfReached()

#define BB CC
#define applyInt BB(int)
#define CC(x) retArgUnclosed

void unbalancedMacros() {
  applyInt );
}

#define expandArgUnclosedCommaExpr(x) (x, clang_analyzer_warnIfReached(), 1
#define f expandArgUnclosedCommaExpr

void unbalancedMacros2() {
  int x =  f(f(1))  )); // note the 'extra' rparens.
}

The code contains no typos, the parens are deliberately not matching at each context.

Besides these bugs, I'm expecting many more - even crashing ones. Might fuzzing could help with resolving the crashes, but that wouldn't help to make it conformant. I'm currently investigating how difficult it would be to use the clang's preprocessor to achieve correct macro expansions. But I suspect that aquiring the state of the PP at the macro expansion location might be difficult.

I've also read some discussion about this on the cfe-dev mailing list as '[cfe-dev] [analyzer] Retrieving macro expansions in the plist output' (http://lists.llvm.org/pipermail/cfe-dev/2018-September/059226.html).

[steakhal]

For the record, my proposed fix is under review at: https://reviews.llvm.org/D93222

[steakhal]

The mentioned godbolt link is not working, use this instead: https://godbolt.org/z/GME4xh

[steakhal]

assigned to @devincoughlin