The text editor is stuttering when loading large files

[ddkwork]

Describe the bug

see title

How to reproduce

xxx.zip

Unzip the zip file is a json file, drag and drop the json file to the window, it takes several seconds to load it on the text editor, and drag the scrollbar after loading it

Example code

package main

import (
    "cogentcore.org/core/goosi/driver/desktop"
    "embed"
    "github.com/ddkwork/golibrary/stream"
    "github.com/go-gl/glfw/v3.3/glfw"

    "cogentcore.org/core/gi"
    "cogentcore.org/core/styles"
    "cogentcore.org/core/texteditor"
)

//go:embed texteditor.go
var samplefile embed.FS

func main() {
    b := gi.NewBody("Cogent Core Text Editor Demo")

    sp := gi.NewSplits(b)
    sp.SetSplits(.5, .5)
    // these are all inherited so we can put them at the top "editor panel" level
    sp.Style(func(s *styles.Style) {
        s.Text.WhiteSpace = styles.WhiteSpacePreWrap
        s.Text.TabSize = 4
        s.Font.Family = string(gi.AppearanceSettings.MonoFont)
    })

    te1 := texteditor.NewEditor(sp)
    te1.Style(func(s *styles.Style) {
        s.Min.X.Ch(20)
        s.Min.Y.Ch(10)
    })
    te2 := texteditor.NewEditor(sp)
    te2.Style(func(s *styles.Style) {
        s.Min.X.Ch(20)
        s.Min.Y.Ch(10)
    })

    tb := texteditor.NewBuf()
    te1.SetBuf(tb)
    te2.SetBuf(tb)

    tb.Hi.Lang = "Go"
    tb.OpenFS(samplefile, "texteditor.go")
    NewWindowRunAndWait(b, func(names []string) {
        s := stream.NewReadFile(names[0])
        tb.SetText(s.Bytes())
        te1.SetBuf(tb)
        te2.SetBuf(tb)
    })

}

func NewWindowRunAndWait(b *gi.Body, DropCallback func(names []string)) {
    w := b.NewWindow().Run()
    if w == nil || w.MainMgr == nil || w.MainMgr.RenderWin == nil {
        return
    }
    win := w.MainMgr.RenderWin.GoosiWin
    ww, ok := win.(*desktop.Window)
    if ok {
        ww.Glw.SetDropCallback(func(w *glfw.Window, names []string) {
            if DropCallback != nil {
                DropCallback(names)
            }
        })
    }
    w.Wait()
}

Relevant output

see title

Platform

Windows

[ddkwork]

[ddkwork]

The GIF situation recorded this time is a little better, and the scroll bar cannot be operated several times before, and sometimes the main window is stuck

[kkoreilly]

I will look into this.

[ddkwork]

Hello, I have conducted several stress tests and the code can be set up within approximately 1.6 seconds. However, after running the main function and dragging and dropping the file to set it up, it takes 6-17 seconds. Which functions should I further stress test?

[kkoreilly]

The cause of the delay is likely the syntax highlighting, which happens in SetFilename, not SetBuf.

[ddkwork]

The cause of the delay is likely the syntax highlighting, which happens in SetFilename, not SetBuf.

Okay, I'm going to read the code carefully and comment out the two aspects of the code to test again, but if that's the case, I think I should stress test the syntax coloring, because it could be caused by one function related to the syntax coloring but not all of it. I think the code editor needs 100% test coverage, write unit tests for all relevant functions, and stress test for all relevant functions, so that we can run the test again after making any subsequent changes, before we are ready to commit the code, as long as the performance of the loading code is optimal, we can rely on various tests to keep it stable.

[kkoreilly]

I agree, and I will write text editor tests in the context of #541.

[kkoreilly]

I will look into this soon.

[ddkwork]

Maybe my bad English translation is inaccurate, but it's actually just this logic: split the file, in the thread, in sections, and the large file only processes the first segment of the split result for the first time. What do you think?

[kkoreilly]

I will consider this idea.

[ddkwork]

Maybe could used github.com/allegro/bigcache

[kkoreilly]

That is an interesting idea that I will consider, but it should not be necessary for the performance improvements for opening files that I will work on soon.

[ddkwork]

Ok

[ddkwork]

I've reimplemented syntax highlighting with unison and have come to the conclusion that:large file loading performance needs to be improved by improving chroma to support streaming reads and also changing to streaming rendering within any gui libs, which is going to take some time to look into, do you know of any other libs that don't use regular expressions to parse tokens other than chroma? Regular expressions don't seem to support streaming mode operations!

[kkoreilly]

Again, as I said above, I am going to improve the performance of the text editor as soon as I have the time. I guarantee you we will have good large file loading performance.