Closed techee closed 3 months ago
When I debug this in SciTE built with Visual C++ 2022 with the AllStyles.toml
, there is a failure at the end of ColouriseTOMLDoc
.
Run-Time Check Failure #2 - Stack around the variable 'escSeq' was corrupted.
This is commonly an array out-of-bounds write. It may be caused by the next issue.
To prevent identifier leakage and potentially bad links static
should be avoided and all of the code, except for the final LexerModule
declaration should be inside an unnamed namespace namespace {
as is done for many other lexers including LexPython.cxx
. After using an unnamed namespace the above problem stopped although that could be due to other changes I have made.
AllStyles.toml
should include examples of styles SCE_TOML_IDENTIFIER
(2) and SCE_TOML_ERROR
(7) if they are possible. Adding the line testlexers.list.styles=1
to SciTE.properties
will show the styles that were produced 0-1 3-6 8-14
.
CharacterSet.h
includes a IsAHeXDigit
which can be used instead of a local IsHexDigit
.
If a background colour is added to SCE_TOML_KEY
(6), like style.toml.6=fore:#000080,bold,back:#FFD0D0
, it shows trailing whitespace as part of the key. It looks weird to me but I'll accept it if that's what you want.
From Visual C++ Code Analysis:
G:\u\hg\pull\lexilla\lexers\LexTOML.cxx(116): warning C26447: The function is declared 'noexcept' but calls function 'GetRelative()' which may throw exceptions (f.6).
G:\u\hg\pull\lexilla\lexers\LexTOML.cxx(134): warning C26447: The function is declared 'noexcept' but calls function 'GetRelative()' which may throw exceptions (f.6).
@techee trailing quotes inside triple-quoted string and trailing space after key were fixed in Notepad4, you can sync the changes.
Fairly sure that a link clash between EscapeSequence
(or its methods) in LexJSON.cxx
and LexTOML.cxx
was the cause of the 'Run-Time Check Failure' since replacing EscapeSequence
with XEscapeSequence
in LexTOML.cxx
fixed it. C++ doesn't allow specifying a struct
is static
so an unnamed namespace should be used.
The g++ crash was with a debug version of GetLineState
which throws for out-of-bounds instead of returning 0.
The g++ crash was with a debug version of
GetLineState
which throws for out-of-bounds instead of returning 0.
It's the difference between Lexilla's TestDocument::GetLineState()
and Scintilla's LineState::GetLineState()
:
int SCI_METHOD TestDocument::GetLineState(Sci_Position line) const {
return lineStates.at(line);
}
int LineState::GetLineState(Sci::Line line) {
if (line < 0)
return 0;
lineStates.EnsureLength(line + 1);
return lineStates[line];
}
@nyamatongwe I believe I (together with fixes from @zufuliu, thanks!) addressed all the issues. Let me know if anything is missing.
Committed in squashed form with a change log item.
@techee I added another commit that fix examples from https://toml.io/en/v1.0.0#keys
fruit.name = "banana" # this is best practice
fruit. color = "yellow" # same as fruit.color
fruit . flavor = "banana" # same as fruit.flavor
though background color for spaces around dot is not fixed.
@techee I added another commit that fix examples from https://toml.io/en/v1.0.0#keys
Thanks, I noticed that problem too but didn't include it into the unit test since based on the documentation this style is discouraged.
I'll add your commit and update the unit test with it.
This patch adds the TOML lexer from the Notepad4 editor originally created by Zufu Liu, see https://github.com/zufuliu/notepad4/issues/806.
Some changes have been made to make it compile and work in Scintilla since Notepad4 contains a modified Scintilla version:
Full diff
```diff --- LexTOML_orig.cxx 2024-08-13 18:35:25.105202586 +0200 +++ LexTOML.cxx 2024-08-13 18:20:05.318285762 +0200 @@ -1,6 +1,10 @@ -// This file is part of Notepad4. -// See License.txt for details about distribution and modification. -//! Lexer for TOML. +// Scintilla source code edit control +/** @file LexTOML.cxx + ** Lexer for TOML language. + **/ +// Based on Zufu Liu's Notepad4 TOML lexer +// Modified for Scintilla by Jiri Techet, 2024 +// The License.txt file describes the conditions under which this software may be distributed. #includeSome of the utility functions were taken from other Notepad4 Scintilla files and changes have been made to adopt the lexer to the Scintilla API but nothing major.
I briefly checked the code of the lexer and things seem to look good in general (the
IsTOMLKey()
function is a bit confusing as it also sets state but I don't know how to name it better or how to rewrite it to avoid duplicated code). Specifically I checked whether insideColouriseTOMLDoc()
the lexer always advances forward to avoid hangups and all the branches seem to be alright.For the test I used the various examples from https://toml.io/en/v1.0.0 - the only slightly incorrect output I noticed was at the end of
and
where the lexer takes the first triple-string delimiters to close the string even though in this case it should probably take the last 3 in the sequence. But I think it's not a big problem in practice and I didn't spend time fixing it.
Please let me know what is missing and what else should be modified to get this merged.