search

dlang-community / D-Scanner

Swiss-army knife for D source code
Boost Software License 1.0
241 stars 80 forks source link

UTFException on dscanner --styleCheck #823

Open zorael opened 3 years ago

zorael commented 3 years ago

Manjaro/Arch x86_64, dscanner-git v0.11.0-1-g2963358 from AUR.

I have a clone of a repo that dscanner --styleCheck throws a UTFException on, and I can't seem to determine why. Making a clean clone (which git claims is identical) makes the issue go away. There are no stray object files in the source tree or anything like that. If there were invalid UTF-8 in the actual source, the new clone would have exhibited the same behaviour.

std.utf.UTFException@/usr/include/dlang/dmd/std/utf.d(1507): Invalid UTF-8 sequence (at index 1)
----------------
??:? pure dchar std.utf.decodeImpl!(true, 0, const(char)[]).decodeImpl(const(char)[], ref ulong) [0x5577ba243fae]
??:? pure @trusted dchar std.utf.decode!(0, immutable(char)[]).decode(ref immutable(char)[], ref ulong) [0x5577ba24420a]
??:? void ddoc.lexer.Lexer.lexWord() [0x5577ba252360]
??:? void ddoc.lexer.Lexer.popFront() [0x5577ba2522e8]
??:? ddoc.sections.Section[] ddoc.sections.splitSections(immutable(char)[]) [0x5577ba255256]
??:? ddoc.comments.Comment ddoc.comments.parseComment(immutable(char)[], immutable(char)[][immutable(char)[]], bool) [0x5577ba250ba9]
??:? bool dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.hasThrowSection(immutable(char)[]) [0x5577ba2b6a46]
??:? void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.FunctionDeclaration)) [0x5577ba2b64a8]
??:? const void dparse.ast.Declaration.accept(dparse.ast.ASTVisitor) [0x5577ba16acea]
??:? void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Declaration)) [0x5577ba2b623b]
??:? const void dparse.ast.StructBody.accept(dparse.ast.ASTVisitor) [0x5577ba1733fc]
??:? void dparse.ast.ASTVisitor.visit(const(dparse.ast.StructBody)) [0x5577ba166780]
??:? const void dparse.ast.StructDeclaration.accept(dparse.ast.ASTVisitor) [0x5577ba17353e]
??:? void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.StructDeclaration)) [0x5577ba2b63a1]
??:? const void dparse.ast.Declaration.accept(dparse.ast.ASTVisitor) [0x5577ba16af0c]
??:? void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Declaration)) [0x5577ba2b623b]
??:? const void dparse.ast.Module.accept(dparse.ast.ASTVisitor) [0x5577ba1701d7]
??:? void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Module)) [0x5577ba2b5d47]
??:? std.container.rbtree.RedBlackTree!(dscanner.analysis.base.Message, " a.line < b.line || (a.line == b.line && a.column < b.column) ", true).RedBlackTree dscanner.analysis.run.analyze(immutable(char)[], const(dparse.ast.Module), const(dscanner.analysis.config.StaticAnalysisConfig), ref dsymbol.modulecache.ModuleCache, const(std.experimental.lexer.TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields; mixin TokenTriviaFields;").TokenStructure)[], bool) [0x5577ba2beb7d]
??:? bool dscanner.analysis.run.analyze(immutable(char)[][], const(dscanner.analysis.config.StaticAnalysisConfig), immutable(char)[], ref dparse.lexer.StringCache, ref dsymbol.modulecache.ModuleCache, bool) [0x5577ba2bb9b6]
??:? _Dmain [0x5577ba319542]

Would it be possible to have it give a helpful error message? At least have it name the file that caused the exception, if not even where in it?

zorael commented 3 years ago

Better backtrace with line numbers.

$ dub run dscanner -- -S --skipTests source
Building package dscanner in /home/zorael/.dub/packages/dscanner-0.11.0/dscanner/
Running pre-generate commands for dscanner...
Performing "debug" build using /usr/bin/dmd for x86_64.
stdx-allocator 2.77.5: target for configuration "library" is up to date.
emsi_containers 0.8.0: target for configuration "library" is up to date.
libdparse 0.17.0: target for configuration "library" is up to date.
dsymbol 0.11.3: target for configuration "library" is up to date.
inifiled 1.3.2: target for configuration "library-quiet" is up to date.
libddoc 0.7.4: target for configuration "lib" is up to date.
dscanner 0.11.0: target for configuration "application" is up to date.
To force a rebuild of up-to-date targets, run again with --force.
Running ../../.dub/packages/dscanner-0.11.0/dscanner/bin/dscanner -S --skipTests source
std.utf.UTFException@/usr/include/dlang/dmd/std/utf.d(1508): Invalid UTF-8 sequence (at index 1)
----------------
/usr/include/dlang/dmd/std/utf.d:1598 pure dchar std.utf.decodeImpl!(true, 0, const(char)[]).decodeImpl(const(char)[], ref ulong) [0x560563303eb9]
/usr/include/dlang/dmd/std/utf.d:1147 pure @trusted dchar std.utf.decode!(0, immutable(char)[]).decode(ref immutable(char)[], ref ulong) [0x5605633044dd]
../../.dub/packages/libddoc-0.7.4/libddoc/src/ddoc/lexer.d:217 void ddoc.lexer.Lexer.lexWord() [0x56056326b443]
../../.dub/packages/libddoc-0.7.4/libddoc/src/ddoc/lexer.d:199 void ddoc.lexer.Lexer.popFront() [0x56056326b397]
../../.dub/packages/libddoc-0.7.4/libddoc/src/ddoc/sections.d:165 ddoc.sections.Section[] ddoc.sections.splitSections(immutable(char)[]) [0x56056326ee28]
../../.dub/packages/libddoc-0.7.4/libddoc/src/ddoc/comments.d:21 ddoc.comments.Comment ddoc.comments.parseComment(immutable(char)[], immutable(char)[][immutable(char)[]], bool) [0x56056326909c]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:284 bool dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.hasThrowSection(immutable(char)[]) [0x560563168c26]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:194 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.FunctionDeclaration)) [0x5605631685c8]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:1274 const void dparse.ast.Declaration.accept(dparse.ast.ASTVisitor) [0x5605632798e3]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:146 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Declaration)) [0x5605631682f4]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:2840 const void dparse.ast.StructBody.accept(dparse.ast.ASTVisitor) [0x5605632842b8]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:321 void dparse.ast.ASTVisitor.visit(const(dparse.ast.StructBody)) [0x560563274200]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:2861 const void dparse.ast.StructDeclaration.accept(dparse.ast.ASTVisitor) [0x560563284470]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:169 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.StructDeclaration)) [0x560563168493]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:1274 const void dparse.ast.Declaration.accept(dparse.ast.ASTVisitor) [0x560563279bca]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:146 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Declaration)) [0x5605631682f4]
../../.dub/packages/libdparse-0.17.0/libdparse/src/dparse/ast.d:2274 const void dparse.ast.Module.accept(dparse.ast.ASTVisitor) [0x560563280374]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/properly_documented_public_functions.d:52 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Module)) [0x560563167ce7]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/run.d:599 std.container.rbtree.RedBlackTree!(dscanner.analysis.base.Message, " a.line < b.line || (a.line == b.line && a.column < b.column) ", true).RedBlackTree dscanner.analysis.run.analyze(immutable(char)[], const(dparse.ast.Module), const(dscanner.analysis.config.StaticAnalysisConfig), ref dsymbol.modulecache.ModuleCache, const(std.experimental.lexer.TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields; mixin TokenTriviaFields;").TokenStructure)[], bool) [0x560563171bc1]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/analysis/run.d:247 bool dscanner.analysis.run.analyze(immutable(char)[][], const(dscanner.analysis.config.StaticAnalysisConfig), immutable(char)[], ref dparse.lexer.StringCache, ref dsymbol.modulecache.ModuleCache, bool) [0x56056316e7ab]
../../.dub/packages/dscanner-0.11.0/dscanner/src/dscanner/main.d:263 _Dmain [0x5605631eab02]
Program exited with code 1
kinke commented 3 years ago

A UTFException is also thrown when using a somewhat more recent D-Scanner version for Phobos (v0.11.0 and a40492bc92e86bad6441fc3aba776fe2c6a5090b both crash when checking ~current Phobos). I bet it's caused by https://github.com/dlang-community/libddoc/commit/7d55a1ada75c5f01899f9b2f5d4cc6d730f6f8aa. Pinging @MoonlightSentinel.

RazvanN7 commented 3 years ago

I verified and it seems that dscanner successfully compiles phobos up to https://github.com/dlang-community/D-Scanner/commit/a40492bc92e86bad6441fc3aba776fe2c6a5090b . So it looks like the failure comes from libdparse after all.

Edit: It looks like the culprit is: https://github.com/dlang-community/libdparse/pull/387