Quality of life improvements

[WebFreak001]

• Token.toString implementation for neater debug printing
• make tokenizer return non-const tokens, so they can be altered afterwards if necessary (no technical reason to make it const)

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 ------ libdparse statistics ------

 statistics (-before, +after)
-library size=3105644 libdparse.a
+library size=3163068 libdparse.a
 rough build time=16s

 ------ DCD statistics ------

 statistics (-before, +after)
 client size=1048544 bin/dcd-client
-server size=3089720 bin/dcd-server
+server size=3089912 bin/dcd-server
 rough build time=75s

-DCD run_tests.sh   Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.23
-DCD run_tests.sh   Maximum resident set size (kbytes): 43260
+DCD run_tests.sh   Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.27
+DCD run_tests.sh   Maximum resident set size (kbytes): 41124

 short requests: (230x)
     min request time =     0.007ms
-    10th percentile  =     0.111ms
-    median time      =     0.402ms
-    90th percentile  =     0.717ms
-    max request time =     1.456ms
+    10th percentile  =     0.116ms
+    median time      =     0.408ms
+    90th percentile  =     0.708ms
+    max request time =     1.422ms

 long requests over 10ms: (51x)
-    min request time =    15.014ms
-    10th percentile  =    15.354ms
-    median time      =    16.416ms
-    90th percentile  =    19.013ms
-    max request time =    26.666ms
+    min request time =    15.295ms
+    10th percentile  =    16.203ms
+    median time      =    17.252ms
+    90th percentile  =    18.977ms
+    max request time =    25.990ms

 top 5 GC sources in server:
 bytes allocated, allocations, type, function, file:line
-       78117888                844 void[] std.array.Appender!(const(TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields,TriviaToken; mixin TokenTriviaFields;"))[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634
+       78109696                843 void[] std.array.Appender!(TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields,TriviaToken; mixin TokenTriviaFields;")[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634
        15257216             476788 void[] core.lifetime._d_newitemT!(Data)._d_newitemT /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/druntime/import/core/lifetime.d:2833
        14316800             164422 void[] std.array.Appender!(DSymbol*[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634
         9773344             305417 Data std.array.Appender!string.Appender.this /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3509
         7372208                281 ubyte[] msgpack.unpacker.Unpacker.InternalBuffer!().initializeBuffer ../../../../.dub/packages/msgpack-d/1.0.4/msgpack-d/src/msgpack/common.d:628

Full build output

``` DUB version 1.34.0, built on Oct 15 2023 LDC - the LLVM D compiler (1.35.0): based on DMD v2.105.2 and LLVM 16.0.6 built with LDC - the LLVM D compiler (1.35.0) Default target: x86_64-unknown-linux-gnu Host CPU: icelake-server http://dlang.org - http://wiki.dlang.org/LDC Registered Targets: aarch64 - AArch64 (little endian) aarch64_32 - AArch64 (little endian ILP32) aarch64_be - AArch64 (big endian) amdgcn - AMD GCN GPUs arm - ARM arm64 - ARM64 (little endian) arm64_32 - ARM64 (little endian ILP32) armeb - ARM (big endian) avr - Atmel AVR Microcontroller bpf - BPF (host endian) bpfeb - BPF (big endian) bpfel - BPF (little endian) hexagon - Hexagon lanai - Lanai loongarch32 - 32-bit LoongArch loongarch64 - 64-bit LoongArch mips - MIPS (32-bit big endian) mips64 - MIPS (64-bit big endian) mips64el - MIPS (64-bit little endian) mipsel - MIPS (32-bit little endian) msp430 - MSP430 [experimental] nvptx - NVIDIA PTX 32-bit nvptx64 - NVIDIA PTX 64-bit ppc32 - PowerPC 32 ppc32le - PowerPC 32 LE ppc64 - PowerPC 64 ppc64le - PowerPC 64 LE r600 - AMD GPUs HD2XXX-HD6XXX riscv32 - 32-bit RISC-V riscv64 - 64-bit RISC-V sparc - Sparc sparcel - Sparc LE sparcv9 - Sparc V9 spirv32 - SPIR-V 32-bit spirv64 - SPIR-V 64-bit systemz - SystemZ thumb - Thumb thumbeb - Thumb (big endian) ve - VE wasm32 - WebAssembly 32-bit wasm64 - WebAssembly 64-bit x86 - 32-bit X86: Pentium-Pro and above x86-64 - 64-bit X86: EM64T and AMD64 xcore - XCore Upgrading project in /home/runner/work/libdparse/libdparse/ Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.35.0/x64/ldc2-1.35.0-linux-x86_64/bin/ldc2 for x86_64. Building libdparse 0.23.2+commit.7.gdb26ddb: building configuration [library] STAT:------ libdparse statistics ------ STAT: STAT:statistics (-before, +after) STAT:library size=3163068 libdparse.a STAT:rough build time=16s STAT: STAT: STAT:------ DCD statistics ------ STAT: { "name": "dcd", "description": "The D Completion Daemon is an auto-complete program for the D programming language", "copyright": "Copyright © 2015-2020, Brian Schott", "authors": [ "Brian Schott" ], "license": "GPL-3.0", "dependencies": { ":dsymbol": "*", "libdparse": {"path":".."}, ":common": "*", "emsi_containers": "~>0.9.0" }, "subPackages": ["dsymbol", "common"], "versions": ["built_with_dub"], "configurations": [ { "name": "library", "targetType": "library", "excludedSourceFiles": [ "src/dcd/client/*", "src/dcd/server/main.d" ] }, { "name": "client", "targetType": "executable", "targetPath": "bin/", "targetName": "dcd-client", "excludedSourceFiles": [ "src/dcd/server/*" ] }, { "name": "server", "targetType": "executable", "targetPath": "bin/", "targetName": "dcd-server", "excludedSourceFiles": [ "src/dcd/client/*" ] } ] } { "fileVersion": 1, "versions": { "dsymbol": "0.14.1", "emsi_containers": "0.9.0", "libdparse": {"path":".."}, "msgpack-d": "1.0.4", "stdx-allocator": "2.77.5" } } STAT:statistics (-before, +after) STAT:client size=1048544 bin/dcd-client STAT:server size=3089912 bin/dcd-server STAT:rough build time=75s STAT: unix:tc001: ... Pass unix:tc002: ... Pass unix:tc003: ... Pass unix:tc004: ... Pass unix:tc005: ... Pass unix:tc006: ... Pass unix:tc007: ... Pass unix:tc008: ... Pass unix:tc009: ... Pass unix:tc010: ... Pass unix:tc011: ... Pass unix:tc012: ... Pass unix:tc013: ... Pass unix:tc014: ... Pass unix:tc015: ... Pass unix:tc016: ... Pass unix:tc017: ... Pass unix:tc018: ... Pass unix:tc019: ... Pass unix:tc020: ... Pass unix:tc021: ... Pass unix:tc022: ... Pass unix:tc023: ... Pass unix:tc024: ... Pass unix:tc025: ... Pass unix:tc026: ... Pass unix:tc027: ... Pass unix:tc028: ... Pass unix:tc029: ... Pass unix:tc030: ... Pass unix:tc031: ... Pass unix:tc032: ... Pass unix:tc033: ... Pass unix:tc034: ... Pass unix:tc035: ... Pass unix:tc036: ... Pass unix:tc037: ... Pass unix:tc038: ... Pass unix:tc039: ... Pass unix:tc040: ... Pass unix:tc041: ... Pass unix:tc042: ... Pass unix:tc043: ... Pass unix:tc044: ... Pass unix:tc045: ... Pass unix:tc046: ... Pass unix:tc047: ... Pass unix:tc048: ... Pass unix:tc049: ... Pass unix:tc050: ... Pass unix:tc051: ... Pass unix:tc052: ... Pass unix:tc053: ... Pass unix:tc054: ... Pass unix:tc055: ... Pass unix:tc056: ... Pass unix:tc057: ... Pass unix:tc058: ... Pass unix:tc059: ... Pass unix:tc060: ... Pass unix:tc061: ... Pass unix:tc062: ... Pass unix:tc620: ... Pass unix:tc717: ... Pass unix:tc_access_modifiers: ... Pass unix:tc_accesschain_type: ... Pass unix:tc_anon_class: ... Pass unix:tc_anon_struct: ... Pass unix:tc_bang_op_or_template: ... Pass unix:tc_base_template_type: ... Pass unix:tc_body_var: ... Pass unix:tc_calltip_in_func: ... Pass unix:tc_char_dot: ... Pass unix:tc_complete_kw: ... Pass unix:tc_ctors: ... Pass unix:tc_currmod_fqn: ... Pass unix:tc_ditto_scopes: ... Pass unix:tc_empty_module: ... Pass 00000unix:tc_empty_requests: ... Pass unix:tc_erroneous_body_content: ... Pass unix:tc_extended_ditto: ... Pass unix:tc_extended_types: ... Pass unix:tc_if_auto_array: ... Pass unix:tc_if_var: ... Pass unix:tc_import_symbol_list: ... Pass unix:tc_incomplete_switch: ... Pass unix:tc_issue558: ... Pass unix:tc_locate_ufcs_function: ... Pass unix:tc_middle_of_utf: ... Pass unix:tc_module_scope_op: ... Pass unix:tc_named_mixin: ... Pass unix:tc_opaque_structs: ... Pass unix:tc_pointer_type_printing: ... Pass unix:tc_pointers: ... Pass unix:tc_recursive_public_import: ... Pass unix:tc_rm_import: ... Pass unix:tc_scope_mess: ... Pass unix:tc_selective_import_list: ... Pass unix:tc_super_scope: ... Pass unix:tc_template_bang_completion: ... Pass unix:tc_template_param_props: ... Pass unix:tc_traits: ... Pass unix:tc_ufcs_alias_this_completion: ... Pass unix:tc_ufcs_array_type_completion: ... Pass unix:tc_ufcs_calltip_in_func: ... Pass unix:tc_ufcs_fundamental_types_completion: ... Pass unix:tc_ufcs_pointer_type_completion: ... Pass unix:tc_ufcs_string_and_string_literal_completion: ... Pass unix:tc_ufcs_struct_completion: ... Pass STAT:DCD run_tests.sh Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.27 STAT:DCD run_tests.sh Maximum resident set size (kbytes): 41124 STAT: STAT:short requests: (230x) STAT: min request time = 0.007ms STAT: 10th percentile = 0.116ms STAT: median time = 0.408ms STAT: 90th percentile = 0.708ms STAT: max request time = 1.422ms STAT: STAT:long requests over 10ms: (51x) STAT: min request time = 15.295ms STAT: 10th percentile = 16.203ms STAT: median time = 17.252ms STAT: 90th percentile = 18.977ms STAT: max request time = 25.990ms STAT: unix:tc001: ... Pass unix:tc002: ... Pass unix:tc003: ... Pass unix:tc004: ... Pass unix:tc005: ... Pass unix:tc006: ... Pass unix:tc007: ... Pass unix:tc008: ... Pass unix:tc009: ... Pass unix:tc010: ... Pass unix:tc011: ... Pass unix:tc012: ... Pass unix:tc013: ... Pass unix:tc014: ... Pass unix:tc015: ... Pass unix:tc016: ... Pass unix:tc017: ... Pass unix:tc018: ... Pass unix:tc019: ... Pass unix:tc020: ... Pass unix:tc021: ... Pass unix:tc022: ... Pass unix:tc023: ... Pass unix:tc024: ... Pass unix:tc025: ... Pass unix:tc026: ... Pass unix:tc027: ... Pass unix:tc028: ... Pass unix:tc029: ... Pass unix:tc030: ... Pass unix:tc031: ... Pass unix:tc032: ... Pass unix:tc033: ... Pass unix:tc034: ... Pass unix:tc035: ... Pass unix:tc036: ... Pass unix:tc037: ... Pass unix:tc038: ... Pass unix:tc039: ... Pass unix:tc040: ... Pass unix:tc041: ... Pass unix:tc042: ... Pass unix:tc043: ... Pass unix:tc044: ... Pass unix:tc045: ... Pass unix:tc046: ... Pass unix:tc047: ... Pass unix:tc048: ... Pass unix:tc049: ... Pass unix:tc050: ... Pass unix:tc051: ... Pass unix:tc052: ... Pass unix:tc053: ... Pass unix:tc054: ... Pass unix:tc055: ... Pass unix:tc056: ... Pass unix:tc057: ... Pass unix:tc058: ... Pass unix:tc059: ... Pass unix:tc060: ... Pass unix:tc061: ... Pass unix:tc062: ... Pass unix:tc620: ... Pass unix:tc717: ... Pass unix:tc_access_modifiers: ... Pass unix:tc_accesschain_type: ... Pass unix:tc_anon_class: ... Pass unix:tc_anon_struct: ... Pass unix:tc_bang_op_or_template: ... Pass unix:tc_base_template_type: ... Pass unix:tc_body_var: ... Pass unix:tc_calltip_in_func: ... Pass unix:tc_char_dot: ... Pass unix:tc_complete_kw: ... Pass unix:tc_ctors: ... Pass unix:tc_currmod_fqn: ... Pass unix:tc_ditto_scopes: ... Pass unix:tc_empty_module: ... Pass 00000unix:tc_empty_requests: ... Pass unix:tc_erroneous_body_content: ... Pass unix:tc_extended_ditto: ... Pass unix:tc_extended_types: ... Pass unix:tc_if_auto_array: ... Pass unix:tc_if_var: ... Pass unix:tc_import_symbol_list: ... Pass unix:tc_incomplete_switch: ... Pass unix:tc_issue558: ... Pass unix:tc_locate_ufcs_function: ... Pass unix:tc_middle_of_utf: ... Pass unix:tc_module_scope_op: ... Pass unix:tc_named_mixin: ... Pass unix:tc_opaque_structs: ... Pass unix:tc_pointer_type_printing: ... Pass unix:tc_pointers: ... Pass unix:tc_recursive_public_import: ... Pass unix:tc_rm_import: ... Pass unix:tc_scope_mess: ... Pass unix:tc_selective_import_list: ... Pass unix:tc_super_scope: ... Pass unix:tc_template_bang_completion: ... Pass unix:tc_template_param_props: ... Pass unix:tc_traits: ... Pass unix:tc_ufcs_alias_this_completion: ... Pass unix:tc_ufcs_array_type_completion: ... Pass unix:tc_ufcs_calltip_in_func: ... Pass unix:tc_ufcs_fundamental_types_completion: ... Pass unix:tc_ufcs_pointer_type_completion: ... Pass unix:tc_ufcs_string_and_string_literal_completion: ... Pass unix:tc_ufcs_struct_completion: ... Pass STAT:top 5 GC sources in server: STAT:bytes allocated, allocations, type, function, file:line STAT: 78109696 843 void[] std.array.Appender!(TokenStructure!(ubyte, "import dparse.lexer:TokenTriviaFields,TriviaToken; mixin TokenTriviaFields;")[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634 STAT: 15257216 476788 void[] core.lifetime._d_newitemT!(Data)._d_newitemT /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/druntime/import/core/lifetime.d:2833 STAT: 14316800 164422 void[] std.array.Appender!(DSymbol*[]).Appender.ensureAddable.__lambda9 /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3634 STAT: 9773344 305417 Data std.array.Appender!string.Appender.this /opt/hostedtoolcache/dc/dmd-2.105.2/x64/dmd2/linux/bin64/../../src/phobos/std/array.d:3509 STAT: 7372208 281 ubyte[] msgpack.unpacker.Unpacker.InternalBuffer!().initializeBuffer ../../../../.dub/packages/msgpack-d/1.0.4/msgpack-d/src/msgpack/common.d:628 ```

[codecov[bot]]

Codecov Report

Merging #509 (8588b70) into master (4a9090d) will not change coverage. The diff coverage is n/a.

Additional details and impacted files

[](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) ```diff @@ Coverage Diff @@ ## master #509 +/- ## ======================================= Coverage 84.25% 84.25% ======================================= Files 11 11 Lines 8561 8561 ======================================= Hits 7213 7213 Misses 1348 1348 ``` | [Files](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) | Coverage Δ | | |---|---|---| | [src/dparse/lexer.d](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community#diff-c3JjL2RwYXJzZS9sZXhlci5k) | `87.85% <ø> (ø)` | | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). Last update [4a9090d...8588b70](https://app.codecov.io/gh/dlang-community/libdparse/pull/509?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dlang-community).