interpolated strings, v2

[WebFreak001]

better implementation, doing all the tokenizing at the start, an interpolated string being multiple tokens

requires no further changes in DCD to support with auto-completion, probably also going to just work with D-Scanner and serve-d

supersedes #508

implementation for DIP1036e

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 ------ libdparse statistics ------

 statistics (-before, +after)
-library size=3163068 libdparse.a
+library size=3197196 libdparse.a
 rough build time=16s

 ------ DCD statistics ------

 statistics (-before, +after)
 client size=1048544 bin/dcd-client
-server size=3089912 bin/dcd-server
+server size=3115384 bin/dcd-server
 rough build time=74s

-DCD run_tests.sh   Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.57
-DCD run_tests.sh   Maximum resident set size (kbytes): 45216
+DCD run_tests.sh   Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.43
+DCD run_tests.sh   Maximum resident set size (kbytes): 41324

 short requests: (230x)
-    min request time =     0.009ms
-    10th percentile  =     0.123ms
-    median time      =     0.431ms
-    90th percentile  =     0.747ms
-    max request time =     2.228ms
+    min request time =     0.013ms
+    10th percentile  =     0.116ms
+    median time      =     0.434ms
+    90th percentile  =     0.753ms
+    max request time =     1.557ms

 long requests over 10ms: (51x)
-    min request time =    17.360ms
-    10th percentile  =    17.650ms
-    median time      =    18.672ms
-    90th percentile  =    21.157ms
-    max request time =    30.142ms
+    min request time =    16.499ms
+    10th percentile  =    17.705ms
+    median time      =    18.483ms
+    90th percentile  =    20.505ms
+    max request time =    29.069ms

 top 5 GC sources in server:
 bytes allocated, allocations, type, function, file:line
        78109696                843 void[] std.array.Appender!(TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields,TriviaToken; mixin TokenTriviaFields;")[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634
        15257216             476788 void[] core.lifetime._d_newitemT!(Data)._d_newitemT /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/druntime/import/core/lifetime.d:2833
        14316800             164422 void[] std.array.Appender!(DSymbol*[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634
         9773344             305417 Data std.array.Appender!string.Appender.this /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3509
         7372208                281 ubyte[] msgpack.unpacker.Unpacker.InternalBuffer!().initializeBuffer ../../../../.dub/packages/msgpack-d/1.0.4/msgpack-d/src/msgpack/common.d:628

Full build output

``` DUB version 1.34.0, built on Oct 15 2023 LDC - the LLVM D compiler (1.35.0): based on DMD v2.105.2 and LLVM 16.0.6 built with LDC - the LLVM D compiler (1.35.0) Default target: x86_64-unknown-linux-gnu Host CPU: skylake-avx512 http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/libdparse/libdparse/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64. Building libdparse 0.23.2+commit.9.ge91e786: building configuration [library] STAT:------ libdparse statistics ------ STAT: STAT:statistics (-before, +after) STAT:library size=3197196 libdparse.a STAT:rough build time=16s STAT: STAT: STAT:------ DCD statistics ------ STAT: { "name": "dcd", "description": "The D Completion Daemon is an auto-complete program for the D programming language", "copyright": "Copyright © 2015-2020, Brian Schott", "authors": [ "Brian Schott" ], "license": "GPL-3.0", "dependencies": { ":dsymbol": "*", "libdparse": {"path":".."}, ":common": "*", "emsi_containers": "~>0.9.0" }, "subPackages": ["dsymbol", "common"], "versions": ["built_with_dub"], "configurations": [ { "name": "library", "targetType": "library", "excludedSourceFiles": [ "src/dcd/client/*", "src/dcd/server/main.d" ] }, { "name": "client", "targetType": "executable", "targetPath": "bin/", "targetName": "dcd-client", "excludedSourceFiles": [ "src/dcd/server/*" ] }, { "name": "server", "targetType": "executable", "targetPath": "bin/", "targetName": "dcd-server", "excludedSourceFiles": [ "src/dcd/client/*" ] } ] } { "fileVersion": 1, "versions": { "dsymbol": "0.14.1", "emsi_containers": "0.9.0", "libdparse": {"path":".."}, "msgpack-d": "1.0.4", "stdx-allocator": "2.77.5" } } STAT:statistics (-before, +after) STAT:client size=1048544 bin/dcd-client STAT:server size=3115384 bin/dcd-server STAT:rough build time=74s STAT: unix:tc001: ... Pass unix:tc002: ... Pass unix:tc003: ... Pass unix:tc004: ... Pass unix:tc005: ... Pass unix:tc006: ... Pass unix:tc007: ... Pass unix:tc008: ... Pass unix:tc009: ... Pass unix:tc010: ... Pass unix:tc011: ... Pass unix:tc012: ... Pass unix:tc013: ... Pass unix:tc014: ... Pass unix:tc015: ... Pass unix:tc016: ... Pass unix:tc017: ... Pass unix:tc018: ... Pass unix:tc019: ... Pass unix:tc020: ... Pass unix:tc021: ... Pass unix:tc022: ... Pass unix:tc023: ... Pass unix:tc024: ... Pass unix:tc025: ... Pass unix:tc026: ... Pass unix:tc027: ... Pass unix:tc028: ... Pass unix:tc029: ... Pass unix:tc030: ... Pass unix:tc031: ... Pass unix:tc032: ... Pass unix:tc033: ... Pass unix:tc034: ... Pass unix:tc035: ... Pass unix:tc036: ... Pass unix:tc037: ... Pass unix:tc038: ... Pass unix:tc039: ... Pass unix:tc040: ... Pass unix:tc041: ... Pass unix:tc042: ... Pass unix:tc043: ... Pass unix:tc044: ... Pass unix:tc045: ... Pass unix:tc046: ... Pass unix:tc047: ... Pass unix:tc048: ... Pass unix:tc049: ... Pass unix:tc050: ... Pass unix:tc051: ... Pass unix:tc052: ... Pass unix:tc053: ... Pass unix:tc054: ... Pass unix:tc055: ... Pass unix:tc056: ... Pass unix:tc057: ... Pass unix:tc058: ... Pass unix:tc059: ... Pass unix:tc060: ... Pass unix:tc061: ... Pass unix:tc062: ... Pass unix:tc620: ... Pass unix:tc717: ... Pass unix:tc_access_modifiers: ... Pass unix:tc_accesschain_type: ... Pass unix:tc_anon_class: ... Pass unix:tc_anon_struct: ... Pass unix:tc_bang_op_or_template: ... Pass unix:tc_base_template_type: ... Pass unix:tc_body_var: ... Pass unix:tc_calltip_in_func: ... Pass unix:tc_char_dot: ... Pass unix:tc_complete_kw: ... Pass unix:tc_ctors: ... Pass unix:tc_currmod_fqn: ... Pass unix:tc_ditto_scopes: ... Pass unix:tc_empty_module: ... Pass 00000unix:tc_empty_requests: ... Pass unix:tc_erroneous_body_content: ... Pass unix:tc_extended_ditto: ... Pass unix:tc_extended_types: ... Pass unix:tc_if_auto_array: ... Pass unix:tc_if_var: ... Pass unix:tc_import_symbol_list: ... Pass unix:tc_incomplete_switch: ... Pass unix:tc_issue558: ... Pass unix:tc_locate_ufcs_function: ... Pass unix:tc_middle_of_utf: ... Pass unix:tc_module_scope_op: ... Pass unix:tc_named_mixin: ... Pass unix:tc_opaque_structs: ... Pass unix:tc_pointer_type_printing: ... Pass unix:tc_pointers: ... Pass unix:tc_recursive_public_import: ... Pass unix:tc_rm_import: ... Pass unix:tc_scope_mess: ... Pass unix:tc_selective_import_list: ... Pass unix:tc_super_scope: ... Pass unix:tc_template_bang_completion: ... Pass unix:tc_template_param_props: ... Pass unix:tc_traits: ... Pass unix:tc_ufcs_alias_this_completion: ... Pass unix:tc_ufcs_array_type_completion: ... Pass unix:tc_ufcs_calltip_in_func: ... Pass unix:tc_ufcs_fundamental_types_completion: ... Pass unix:tc_ufcs_pointer_type_completion: ... Pass unix:tc_ufcs_string_and_string_literal_completion: ... Pass unix:tc_ufcs_struct_completion: ... Pass STAT:DCD run_tests.sh Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.43 STAT:DCD run_tests.sh Maximum resident set size (kbytes): 41324 STAT: STAT:short requests: (230x) STAT: min request time = 0.013ms STAT: 10th percentile = 0.116ms STAT: median time = 0.434ms STAT: 90th percentile = 0.753ms STAT: max request time = 1.557ms STAT: STAT:long requests over 10ms: (51x) STAT: min request time = 16.499ms STAT: 10th percentile = 17.705ms STAT: median time = 18.483ms STAT: 90th percentile = 20.505ms STAT: max request time = 29.069ms STAT: unix:tc001: ... Pass unix:tc002: ... Pass unix:tc003: ... Pass unix:tc004: ... Pass unix:tc005: ... Pass unix:tc006: ... Pass unix:tc007: ... Pass unix:tc008: ... Pass unix:tc009: ... Pass unix:tc010: ... Pass unix:tc011: ... Pass unix:tc012: ... Pass unix:tc013: ... Pass unix:tc014: ... Pass unix:tc015: ... Pass unix:tc016: ... Pass unix:tc017: ... Pass unix:tc018: ... Pass unix:tc019: ... Pass unix:tc020: ... Pass unix:tc021: ... Pass unix:tc022: ... Pass unix:tc023: ... Pass unix:tc024: ... Pass unix:tc025: ... Pass unix:tc026: ... Pass unix:tc027: ... Pass unix:tc028: ... Pass unix:tc029: ... Pass unix:tc030: ... Pass unix:tc031: ... Pass unix:tc032: ... Pass unix:tc033: ... Pass unix:tc034: ... Pass unix:tc035: ... Pass unix:tc036: ... Pass unix:tc037: ... Pass unix:tc038: ... Pass unix:tc039: ... Pass unix:tc040: ... Pass unix:tc041: ... Pass unix:tc042: ... Pass unix:tc043: ... Pass unix:tc044: ... Pass unix:tc045: ... Pass unix:tc046: ... Pass unix:tc047: ... Pass unix:tc048: ... Pass unix:tc049: ... Pass unix:tc050: ... Pass unix:tc051: ... Pass unix:tc052: ... Pass unix:tc053: ... Pass unix:tc054: ... Pass unix:tc055: ... Pass unix:tc056: ... Pass unix:tc057: ... Pass unix:tc058: ... Pass unix:tc059: ... Pass unix:tc060: ... Pass unix:tc061: ... Pass unix:tc062: ... Pass unix:tc620: ... Pass unix:tc717: ... Pass unix:tc_access_modifiers: ... Pass unix:tc_accesschain_type: ... Pass unix:tc_anon_class: ... Pass unix:tc_anon_struct: ... Pass unix:tc_bang_op_or_template: ... Pass unix:tc_base_template_type: ... Pass unix:tc_body_var: ... Pass unix:tc_calltip_in_func: ... Pass unix:tc_char_dot: ... Pass unix:tc_complete_kw: ... Pass unix:tc_ctors: ... Pass unix:tc_currmod_fqn: ... Pass unix:tc_ditto_scopes: ... Pass unix:tc_empty_module: ... Pass 00000unix:tc_empty_requests: ... Pass unix:tc_erroneous_body_content: ... Pass unix:tc_extended_ditto: ... Pass unix:tc_extended_types: ... Pass unix:tc_if_auto_array: ... Pass unix:tc_if_var: ... Pass unix:tc_import_symbol_list: ... Pass unix:tc_incomplete_switch: ... Pass unix:tc_issue558: ... Pass unix:tc_locate_ufcs_function: ... Pass unix:tc_middle_of_utf: ... Pass unix:tc_module_scope_op: ... Pass unix:tc_named_mixin: ... Pass unix:tc_opaque_structs: ... Pass unix:tc_pointer_type_printing: ... Pass unix:tc_pointers: ... Pass unix:tc_recursive_public_import: ... Pass unix:tc_rm_import: ... Pass unix:tc_scope_mess: ... Pass unix:tc_selective_import_list: ... Pass unix:tc_super_scope: ... Pass unix:tc_template_bang_completion: ... Pass unix:tc_template_param_props: ... Pass unix:tc_traits: ... Pass unix:tc_ufcs_alias_this_completion: ... Pass unix:tc_ufcs_array_type_completion: ... Pass unix:tc_ufcs_calltip_in_func: ... Pass unix:tc_ufcs_fundamental_types_completion: ... Pass unix:tc_ufcs_pointer_type_completion: ... Pass unix:tc_ufcs_string_and_string_literal_completion: ... Pass unix:tc_ufcs_struct_completion: ... Pass STAT:top 5 GC sources in server: STAT:bytes allocated, allocations, type, function, file:line STAT: 78109696 843 void[] std.array.Appender!(TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields,TriviaToken; mixin TokenTriviaFields;")[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634 STAT: 15257216 476788 void[] core.lifetime._d_newitemT!(Data)._d_newitemT /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/druntime/import/core/lifetime.d:2833 STAT: 14316800 164422 void[] std.array.Appender!(DSymbol*[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634 STAT: 9773344 305417 Data std.array.Appender!string.Appender.this /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3509 STAT: 7372208 281 ubyte[] msgpack.unpacker.Unpacker.InternalBuffer!().initializeBuffer ../../../../.dub/packages/msgpack-d/1.0.4/msgpack-d/src/msgpack/common.d:628 ```

[codecov[bot]]

Codecov Report

Merging #510 (bdd87a6) into master (63556f0) will increase coverage by 0.08%. The diff coverage is 88.00%.

Additional details and impacted files

[](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) ```diff @@ Coverage Diff @@ ## master #510 +/- ## ========================================== + Coverage 84.25% 84.34% +0.08% ========================================== Files 11 11 Lines 8561 8776 +215 ========================================== + Hits 7213 7402 +189 - Misses 1348 1374 +26 ``` | [Files](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) | Coverage Δ | | |---|---|---| | [src/dparse/astprinter.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9hc3RwcmludGVyLmQ=) | `95.00% <100.00%> (+0.05%)` | :arrow_up: | | [src/dparse/formatter.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9mb3JtYXR0ZXIuZA==) | `48.85% <100.00%> (+0.66%)` | :arrow_up: | | [src/dparse/parser.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9wYXJzZXIuZA==) | `92.66% <86.11%> (-0.07%)` | :arrow_down: | | [src/dparse/lexer.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9sZXhlci5k) | `88.47% <93.38%> (+0.62%)` | :arrow_up: | | [src/dparse/ast.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9hc3QuZA==) | `68.97% <53.57%> (-0.72%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/dlang-community/libdparse/pull/510/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). Last update [63556f0...bdd87a6](https://app.codecov.io/gh/dlang-community/libdparse/pull/510?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community).

[schveiguy]

I'm all for merging this. I don't really have a good feel for what is the proper way to do these things, but we definitely need something. As noted from the linked bug reports, people are finding the lack of istring support troublesome, and it will only get worse. I think it is better to have a buggy or substandard implementation than no attempt.

Needs a rebase though...

[rikkimax]

Same idea as in the compiler.

The lexer has a mini parser that consumes tokens and spits out others.

It at least lines up.