add `:common` subpackage without libdparse

[WebFreak001]

adds ddoc.unhighlight and ddoc.types modules for more useful usage. Unhighlight can be used to emit embedded D code as-is, instead of parsing it and (potentially) causing issues as well as needing a dependency on libdparse.

The ddoc.types module now contains the Comment struct that was in ddoc.comments and has a new parseUnexpanded and parse call, which does what parseComment has done before, but with unhighlight, without expanding, which is commonly used for example in D-Scanner to just check if a comment is ditto or contains some section with a given name.

With only minor modifications to D-Scanner just the libddoc:common subpackage can be used there, avoiding keeping libddoc libdparse up-to-date and synced.

Makes the root libdparse version any version up to <1.0.0, because only tokenization is used and updates should not be needed that often. Having the separate common package makes it less likely that things will break for packages not needing libdparse, other packages will manually specify versions that will make ddoc use older versions if necessary.

[RazvanN7]

Thank you for doing this!

[RazvanN7]

@WebFreak001 This seems to have broken something in libbdoc. I tried using the newest version of dscanner (that contains the version of libddoc with this PR), but ended up with an error [1]:

std.utf.UTFException@/home/razvan/dmd/dmd/generated/linux/release/64/../../../../../phobos/std/utf.d(1498): Invalid UTF-8 sequence (at index 1)
----------------
/home/razvan/dmd/dmd/generated/linux/release/64/../../../../../phobos/std/utf.d:1588 pure dchar std.utf.decodeImpl!(true, 0, const(char)[]).decodeImpl(const(char)[], ref ulong) [0x55ce6c186a5d]
/home/razvan/dmd/dmd/generated/linux/release/64/../../../../../phobos/std/utf.d:1137 pure @trusted dchar std.utf.decode!(0, immutable(char)[]).decode(ref immutable(char)[], ref ulong) [0x55ce6c186979]
libddoc/common/source/ddoc/lexer.d:217 void ddoc.lexer.Lexer.lexWord() [0x55ce6c278c6b]
libddoc/common/source/ddoc/lexer.d:199 void ddoc.lexer.Lexer.popFront() [0x55ce6c278bbe]
libddoc/common/source/ddoc/sections.d:165 ddoc.sections.Section[] ddoc.sections.splitSections(immutable(char)[]) [0x55ce6c27c934]
libddoc/common/source/ddoc/types.d:50 ddoc.types.Comment ddoc.types.Comment.parse(immutable(char)[], immutable(char)[][immutable(char)[]], bool, immutable(char)[] delegate(immutable(char)[])) [0x55ce6c276c14]
libddoc/common/source/ddoc/types.d:36 ddoc.types.Comment ddoc.types.Comment.parse(immutable(char)[], immutable(char)[][immutable(char)[]], bool, immutable(char)[] function(immutable(char)[])*) [0x55ce6c276bd7]
libddoc/src/ddoc/comments.d:21 ddoc.types.Comment ddoc.comments.parseComment(immutable(char)[], immutable(char)[][immutable(char)[]], bool) [0x55ce6c276434]
src/dscanner/analysis/properly_documented_public_functions.d:295 const(ddoc.types.Comment) dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.setLastDdocParams(ulong, ulong, immutable(char)[]) [0x55ce6c362c9c]
src/dscanner/analysis/properly_documented_public_functions.d:167 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.StructDeclaration)) [0x55ce6c362464]
libdparse/src/dparse/ast.d:1288 const void dparse.ast.Declaration.accept(dparse.ast.ASTVisitor) [0x55ce6c211a70]
src/dscanner/analysis/properly_documented_public_functions.d:146 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Declaration)) [0x55ce6c3622f4]
libdparse/src/dparse/ast.d:2289 const void dparse.ast.Module.accept(dparse.ast.ASTVisitor) [0x55ce6c217f4c]
src/dscanner/analysis/properly_documented_public_functions.d:52 void dscanner.analysis.properly_documented_public_functions.ProperlyDocumentedPublicFunctions.visit(const(dparse.ast.Module)) [0x55ce6c361ce7]
src/dscanner/analysis/run.d:599 std.container.rbtree.RedBlackTree!(dscanner.analysis.base.Message, " a.line < b.line || (a.line == b.line && a.column < b.column) ", true).RedBlackTree dscanner.analysis.run.analyze(immutable(char)[], const(dparse.ast.Module), const(dscanner.analysis.config.StaticAnalysisConfig), ref dsymbol.modulecache.ModuleCache, const(std.experimental.lexer.TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields; mixin TokenTriviaFields;").TokenStructure)[], bool) [0x55ce6c315c09]
src/dscanner/analysis/run.d:247 bool dscanner.analysis.run.analyze(immutable(char)[][], const(dscanner.analysis.config.StaticAnalysisConfig), immutable(char)[], ref dparse.lexer.StringCache, ref dsymbol.modulecache.ModuleCache, bool) [0x55ce6c3127f3]
src/dscanner/main.d:263 _Dmain [0x55ce6c3946b4]

I think that the breakage comes from here:

libddoc/common/source/ddoc/types.d:50 ddoc.types.Comment ddoc.types.Comment.parse(immutable(char)[], immutable(char)[][immutable(char)[]], bool, immutable(char)[] delegate(immutable(char)[])) [0x55ce6c276c14]
libddoc/common/source/ddoc/types.d:36 ddoc.types.Comment ddoc.types.Comment.parse(immutable(char)[], immutable(char)[][immutable(char)[]], bool, immutable(char)[] function(immutable(char)[])*) [0x55ce6c276bd7]
libddoc/src/ddoc/comments.d:21 ddoc.types.Comment ddoc.comments.parseComment(immutable(char)[], immutable(char)[][immutable(char)[]], bool) [0x55ce6c276434]

Any ideas for a quick fix?

[1] https://github.com/dlang/phobos/pull/8136

[WebFreak001]

hm no logic has changed though, all the old functions should still have the same execution. Do you know which file this happens on?

[RazvanN7]

yes, it is etc/c/sqlite3.d from phobos (literally the first file it encounters in etc). I tried running dscanner on std and it successfully analyzes some files before it fails on std/utf.d.

[WebFreak001]

can't reproduce locally (issue with GC deallocating when memory is limited?)

[RazvanN7]

Have you tried running dscanner (commit: 2400d9c9e7763c264412048021d0bd8f977d7e0c) on phobos? (via make -f posix.mak dscanner). This fails on both my machine and the phobos testing pipeline

[WebFreak001]

oh I'll try that later, I tried ./bin/dscanner -S phobos with phobos checked out which works fine.

[RazvanN7]

I investigated this issue and this PR isn't the culprit. The issue might be with dscanner after all.

[RazvanN7]

It turns out dscanner starts failing phobos with the UTFException since: https://github.com/dlang-community/D-Scanner/commit/a40492bc92e86bad6441fc3aba776fe2c6a5090b . It looks like the libdparse upgrade introduced the regression.