search

XAMPPRocky / tokei

Count your code, quickly.
Other
10.51k stars 504 forks source link

Incorrect code line count caused by ChildLanguage #1073

Open LeoQ7 opened 3 months ago

LeoQ7 commented 3 months ago 
const demo: &str = "Hello, World!";

/// markdown? docstring?

Output:

> tokei poc.rs
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1            0            0            0            0
 |- Markdown             1            1            0            1            0
 (Total)                              1            0            1            0
===============================================================================
 Total                   1            0            0            0            0
===============================================================================

Trace:

[2024-03-06T18:24:01Z TRACE tokei::language::language_type] const demo: &str = "Hello, World!";
[2024-03-06T18:24:01Z TRACE tokei::language::syntax] Start "\""
[2024-03-06T18:24:01Z TRACE tokei::language::syntax] End "\""
[2024-03-06T18:24:01Z TRACE tokei::language::syntax]

[2024-03-06T18:24:01Z TRACE tokei::language::syntax]

[2024-03-06T18:24:01Z TRACE tokei::language::syntax] /// markdown? docstring?
[2024-03-06T18:24:01Z TRACE tokei::language::syntax] Markdown found: " markdown? docstring?"
[2024-03-06T18:24:01Z TRACE tokei::language::language_type] markdown? docstring?
[2024-03-06T18:24:01Z TRACE tokei::language::syntax] ^ Skippable
[2024-03-06T18:24:01Z TRACE tokei::language::syntax] Comment No.1

Expected output:

> tokei poc.rs
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1            3            1            1            1
 (Total)                              3            1            1            1
===============================================================================
 Total                   1            3            1            1            1
===============================================================================
XAMPPRocky commented 3 months ago

Thank you for your issue! However tokei is correct here. All doc comments in Rust are markdown and this code is invalid Rust because it needs to be attached to an item.

LeoQ7 commented 3 months ago

Thank you for the clarification!

For the following code snippet, is the result from tokei also correct?

const demo: &str = "Hello, World!";

/// markdown? docstring?
const demo1: &str = "Hello, World!";

Output:

[2024-03-07T04:45:50Z TRACE tokei::language::language_type] const demo: &str = "Hello, World!";
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] Start "\""
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] End "\""
[2024-03-07T04:45:50Z TRACE tokei::language::syntax]

[2024-03-07T04:45:50Z TRACE tokei::language::syntax]

[2024-03-07T04:45:50Z TRACE tokei::language::syntax] /// markdown? docstring?

[2024-03-07T04:45:50Z TRACE tokei::language::syntax] Markdown found: " markdown? docstring?\n"
[2024-03-07T04:45:50Z TRACE tokei::language::language_type] markdown? docstring?
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] ^ Skippable
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] Comment No.1
[2024-03-07T04:45:50Z TRACE tokei::language::language_type] const demo1: &str = "Hello, World!";
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] Start "\""
[2024-03-07T04:45:50Z TRACE tokei::language::syntax] End "\""
[2024-03-07T04:45:50Z TRACE tokei::language::language_type] const demo1: &str = "Hello, World!";
[2024-03-07T04:45:50Z TRACE tokei::language::language_type] Code No.1
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1            1            1            0            0
 |- Markdown             1            1            0            1            0
 (Total)                              2            1            1            0
===============================================================================
 Total                   1            1            1            0            0
===============================================================================

IMHO, LoC should be 2 here?

A correct example is: 

const demo: u8 = 42;

/// markdown? docstring?
const demo1: &str = "Hello, World!";

Output:

[2024-03-07T06:18:40Z TRACE tokei::language::language_type] Using Simple Parse on "const demo: u8 = 42;\n\n"
[2024-03-07T06:18:40Z TRACE tokei::language::language_type] /// markdown? docstring?
[2024-03-07T06:18:40Z TRACE tokei::language::syntax] /// markdown? docstring?

[2024-03-07T06:18:40Z TRACE tokei::language::syntax] Markdown found: " markdown? docstring?\n"
[2024-03-07T06:18:40Z TRACE tokei::language::language_type] markdown? docstring?
[2024-03-07T06:18:40Z TRACE tokei::language::syntax] ^ Skippable
[2024-03-07T06:18:40Z TRACE tokei::language::syntax] Comment No.1
[2024-03-07T06:18:40Z TRACE tokei::language::language_type] const demo1: &str = "Hello, World!";
[2024-03-07T06:18:40Z TRACE tokei::language::syntax] Start "\""
[2024-03-07T06:18:40Z TRACE tokei::language::syntax] End "\""
[2024-03-07T06:18:40Z TRACE tokei::language::language_type] const demo1: &str = "Hello, World!";
[2024-03-07T06:18:40Z TRACE tokei::language::language_type] Code No.1
===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Rust                    1            3            2            0            1
 |- Markdown             1            1            0            1            0
 (Total)                              4            2            1            1
===============================================================================
 Total                   1            3            2            0            1
===============================================================================
LeoQ7 commented 3 months ago

A similar issue is #713 , my guess is that the early return at https://github.com/XAMPPRocky/tokei/blob/master/src/language/syntax.rs#L250-L252 somehow affected the counting of previous lines when there is a quote.