Improve performance of text highlighting

[tdauth]

https://stackoverflow.com/questions/70815376/slow-formatting-in-qtextedit

[tdauth]

Scanner needs improvement:

• Scan for string literals.
• Scan for array boundaries.
• function calls ()
• Add all JASS keywords

[tdauth]

Especially common.j needs long. It does not have string literals or function bodies with identifiers. Maybe we have to prestore stuff??

[tdauth]

Even with string literals and keywords and brackets it takes the same time:

Ending highlighting tokens with tokens size: 46852 and elapsed time 75733 ms and in seconds 75 and in minutes 1

Custom scripts are much faster.

[tdauth]

Maybe it is painted directly triggering some event? I could try to disable this. Maybe the moving through the document is wrong.

[tdauth]

It might because of the wrong order of tokens/syntax errors. The syntax errors were in the wrong order. They all have to be ordered by their occurance in the document!

[tdauth]

Better idea: Just highlight the lines visible in the viewport. This will reduce the highlighting from like 4000 lines to 100 or something. However, it needs constant updating but only once. After that it is stored until the text changes. The highlighting in Kate takes about 2 seconds and it doesn't seem to only highlight the visible area.

[tdauth]

Parse Errors are still not ordered by their location. Order them before highlighting. Do the same for tokens. Merge both highlightings together into one list with information for highlighting.

They could use some shared parent type.

[tdauth]

• Check if highlighting stays the same when inserting or removing lines in a QTextEdit.
• If yes try to keep the previous highlighting info and create a delta only for added or changed text blocks.
• Store all parse errors as strings woth line Numbers and columns in the highlighting info. The main window GUI can directly add the string list. Check how long this operation takes and if a delta is required here.
• Serialize highlighting info with a hash of the File content and load it for a newly opened file. Use https://doc.qt.io/qt-5/qcryptographichash.html#details and store the highlighting info in memory and files in the parser/scanner thread every X seconds and on closing the application. Load them on opening it. They should be limited by the size of the files and not the number of files. Maybe cleanup unused ones after some weeks or days or even hours/minutes if they are on the disk and do not match anything anymore.

[tdauth]

Use QPlainTextEdit and block signals. See https://stackoverflow.com/questions/70815376/slow-formatting-in-qtextedit?noredirect=1#comment125216720_70815376

[tdauth]

https://stackoverflow.com/questions/17466046/qtextedit-vs-qplaintextedit

[tdauth]

Diffs of highlighting infos:

Go through all lines and all columns:

• Element is new (added) until with find an the old highlighting again.
• Element does not exist anymore (removed). The removed highlightings can be discarded.

Only these two possibilities exist.

function test takes unit whichUnit returns nothing
endfunction

is changed to

// comment
function test takes nothing returns nothing 
endfunction

generates the following diff:

+ comment (0, 0)
+ keyword (1,19)

Hence, it only applies two highlighting changes.

Generating diffs would also allow to highlight a document step by step, highlighting blocks.

[tdauth]

Ending highlighting code elements with elements size: 21456 and elapsed time 31435 ms and in seconds 31 and in minutes 0

[tdauth]

QSyntaxHighlighter helped.