AvalonEdit issues with large files

[zaksmolen]

I am trying to use AvalonEdit integrated in my application. We have the potential for some obscenely large text files (115MB, 1 million+ lines). That is too large to put into a string, so the normal TextEditor.Load(string filename) function fails. After researching and finding a few somewhat-related threads I put this together:

private static IEnumerable<char> getFileAsIEnumerable(string fname)
{
    using (StreamReader file = new StreamReader(fname))
    {
        while (file.Peek() >= 0)
        {
            yield return (char)file.Read();
        }
    }
}       

// in other part of the code
EditorBox.Document = new TextDocument(getFileAsIEnumerable(fname));

I tried creating a char Rope and passing that into a text source and such, but that was actually consistently a tiny bit slower. As a benchmark I also tried loading this file in SharpDevelop and it loads in half the time that it takes to run the above code (<=5 seconds instead of 10). Does anyone have any ideas as to what could help speed it up to match the performance seen in SharpDevelop?

A few seconds doesn't sound too bad, but a. it's causing the application to hang and b. there are also a few seconds for other calculations, such as pre-loading the syntax coloring (otherwise scrolling the 1MM lines is jumpy) and other parsing we have to do with the file contents. So, now the application hangs for at least 15 seconds total (10 for loading and 5 for parsing and such). And that is on a better than average machine. Hoping to reduce loading time as much as possible to help make things run smoother.

[jogibear9988]

Maybe look at https://www.codeproject.com/articles/161871/fast-colored-textbox-for-syntax-highlighting It has a lazy load mode directly build in. You can use it in WPF with WinFormsHost Control

[zaksmolen]

Thanks for the fast feedback. That is a pretty awesome control, and even has some things I had to manually implement myself in my application using AvalonEdit. Unfortunately we already put a bunch of time and effort into porting from a previous control to AvalonEdit and aren't going to ditch it and redo everything with a new control again.

I like the idea of lazy loading, but it may have some trouble with our usage. In the case where people use million-line files, they will be searching and jumping around quite a lot. That would be tough on a lazy-load system.

I'm more curious why just the file loading on my application (not much else going on in the background) takes consistently 10 seconds while SharpDevelop loading the same file took 5 or less. Both tests were done 10+ times with consistent results so it wasn't just some fluke. Ideally, I would like to do whatever is done in SD. I have been reading through the source but nothing is jumping out at me. It almost even looks like they read the whole file in as a string, but that can't work because the file contents are too long for a string (again, the guiding motivation for writing my own loading functions).