Think about how to handle large files

[pythonesque]

We will want to support a "read only" mode for this and avoid loading the entire file into memory. We will also want to do certain things asynchronously, like figure out line count, so as not to block editing. We may want to consider allowing asynchronous save as well (unfortunately, I don't think there's any way to significantly speed up saving a very large file in the general case, unless you get very lucky with block size and have control over the filesystem).

[gchp]

Yeah, I was thinking about this the other day. If we are to only load chunks of files at a time, can we still do things like search, line count, char count etc? We will need to decide what is considered a large file, so we can impose these kind of restrictions on anything above that size.

[gchp]

We will also have to change how files are saved to accommodate large files too. Right now files are saved to a temporary directory then moved to replace the old version. This won't work well for large files.

[ellisadigvom]

For a data structure, vis uses a piece table, to store the file in memory. This is very efficient for large files, since you can just mmap the file into memory.

Also, you might want to read this paper.

[crespyl]

A piece table might be a nice half-way step towards a rope, which I think would, in principle, also support mmap-ing files into a giant leaf node.

Thanks for that paper @ellisadigvom, this stuff is really helpful for someone as inexperienced as myself!

[gchp]

@ellisadigvom that's very interesting, thanks for sharing! I have been leaning quite heavily towards a Rope recently, but have not had the time to sit down and try it out. This looks like a good alternative, though, definitely worth looking into. I had actually never heard of a piece table before..

Gonna give vis a shot and see how it handles some big files :)

[gchp]

So, anyone know of a piece table implementation in rust? :)