Huge memory consumption for file with millions of lines

[VaGNaroK]

When opening a backup .json file of my telegram conversations, with xed, in a short time I noticed my pc crashing, without understanding why I decided to open the system monitor and I face it:

https://postimg.cc/s1JZpRY0

https://postimg.cc/ZBx0Frdy

https://postimg.cc/PPxNYvrP

I don't know if to open a file of just over 200 mb it uses all this memory. I uploaded the file for you to test.

https://mega.nz/file/l4BUVJ6b#2lZ-P2VqBkgsRfY6bgC0Er_GOm2fjRmLpGw4ApYQPA0

[xenopeek]

It's reproducible on Linux Mint 20 so this isn't a 20.1 beta issue. I could reproduce it with another 200MB file with just "hello world" on each line so it's not specific to your file or to JSON format. Your file had about 7.5 million lines of text. I think that is the trouble here.

I could reproduce the same with Geany, Gedit, Kate, Micro (console text editor), Mousepad, Pluma and Sublime Text — before I could kill these they also used upwards of 1.5 GB RAM and that was with the file not entirely loaded yet. The only text editors I tried that it worked were Vim (console text editor), GVim (basically Vim console text editor wrapped with a GUI) and Nano (console text editor). Vim (and GVim) were most memory efficient, using about 280 MB memory. Nano used about 680 MB memory.

As such I don't think this is a bug but stems from how Xed (and other text editors) loads the entire file into memory and how it organizes each line of text in memory using much more memory that we would expect. Perhaps that can be improved upon, IDK. That so many other editors have the same trouble makes me think this may be harder to do something about.

[nerkn]

May xed ask if we really want to open this kind of file, before processing, or may be pop a notification giving option like "want to abort loading file" after x seconds of processing?

[olyerickson]

I don't believe this is a problem related to huge files --- although that would most certainly cause a problem!

I've experienced similar problems with:

• xed 2.4.2
• Linux Mint 19.3
• MATE

My problem with runaway memory is trivially easy to replicate:

• Open new document in xed (NOTE: Easily reproduced!!)
• Start typing
• Continue typing without saving
• xed will slow.....down........and your fan will actually kick in (Like you're editing a video!)
• Save out the file and everything is back to normal; normal speed, no fan!

I'm running on a Lenovo P51 with SSD; I'm not used to a text editor with nothing in it being s-l-o-w!

[OdinVex]

EmEditor has no trouble with things like this. I used it often when I was running Windows, and I'd edit 60 GB+ files sometimes. Granted working with larger files is probably rare and unique, but I have come to desperately wish EmEditor ran on Linux for lack of any editor for Linux being (in my opinion) decent about working with large files or files needing syntactical highlighting (try editing a 100 MB file in Xed with syntax-highlighting...ugh).

[CO-Windler]

May be you need a disk editor with selectable hex and ASCII mode to edit really huge files in "disk image" mode or such.

Most normal text editors attempt to copy the entire file into ram and so eventually clog up swap space and cause thrashing, which may even damage SSD drives by excessive write access. Thus a warning message popup when loading huge files is a good idea.

[OdinVex]

Almost all Linux text editors really bite because they're inefficient at editing large files, just as @CO-Windler mentioned. Just see my comment on EmEditor (not that I'm advocating for it, they did away with Lifetime licenses, screw that). It proves that it can be done. I can edit multi-GB files with no issues in EmEditor. Supposedly works on Wine, but I don't like using non-native stuff.