Large file support

[watsonian]

So, I use my text editor for all kinds of things (writing code, reading readmes, editing dotfiles, etc). One of the things I do most often during any given day is open up large log files to troubleshoot Enterprise problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately -- the file selector didn't change and the entire window went completely white 4 seconds after I tried viewing the file. I could still close the top level window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice progress indicator while it was loading:

Once it loaded it was as responsive as any other file I load (scrolled fast, normal text highlight performance, etc). It would be nice if we set this as a baseline for expected behavior (progress bar + responsive after loading).

[nathansobo]

Yeah, we currently do a blocking read to load files. We need to switch to something event-driven. I also wonder if we can load the file progressively off the disk as needed so we don't have to seek the whole thing into memory. But that's super-advanced. Just avoiding blocking the UI thread would be a good start. Yet another thing I'd like to do with Node.

On Sun, Feb 24, 2013 at 5:54 PM, Joel Watson notifications@github.comwrote:

So, I use my text editor for all kinds of things (writing code, reading readmes, editing dotfiles, etc). One of the things I do most often during any given day is open up large log files to troubleshoot Enterprise problems. These files can sometimes get as large as 500-800MB.

I just tried viewing a 350MB log file and Atom locked up immediately -- the file selector didn't change and the entire window went completely white 4 seconds after I tried viewing the file. I could still close the top level window, but we should still have better handling for this kind of thing.

Sublime Text 2 took about 55 seconds to open the file, but gave a nice progress indicator while it was loading:

[image: Screen Shot 2013-02-24 at 4 50 56 PM]https://f.cloud.github.com/assets/244/190379/c0abc27e-7ee5-11e2-8ec4-4205cdee0ad0.png

Once it loaded it was as responsive as any other file I load (scrolled fast, normal text highlight performance, etc). It would be nice if we set this as a baseline for expected behavior (progress bar + responsive after loading).

— Reply to this email directly or view it on GitHubhttps://github.com/github/atom/issues/307.

[watsonian]

I just tried opening a 124MB JSON file directly from the CLI and got this:

RangeError: Maximum call stack size exceeded
    at BufferChangeOperation.module.exports.BufferChangeOperation.changeBuffer (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:99:17)
    at BufferChangeOperation.module.exports.BufferChangeOperation.do (/Applications/Atom.app/Contents/Resources/src/app/buffer-change-operation.js:37:19)
    at Buffer.module.exports.Buffer.pushOperation (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:348:31)
    at Buffer.module.exports.Buffer.change (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:314:20)
    at Buffer.module.exports.Buffer.setText (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:174:19)
    at Buffer.module.exports.Buffer.reload (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:126:12)
    at new Buffer (/Applications/Atom.app/Contents/Resources/src/app/buffer.js:65:14)
    at Project.module.exports.Project.buildBuffer (/Applications/Atom.app/Contents/Resources/src/app/project.js:270:16)
    at Project.module.exports.Project.bufferForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:259:33)
    at Project.module.exports.Project.buildEditSessionForPath (/Applications/Atom.app/Contents/Resources/src/app/project.js:180:41) index.html:23
window.onload

[probablycorey]

This moved forward after #939. But there is still some telepath work to get the rest working. Adding single-user mode to telepath that eliminates the need for each character to have a location may be a solution.

[stuart-warren]

I'm failing to open up a 1.4MB HTML file with the same RangeError: Maximum call stack exceeded

Running Version 0.105 Has anything happened on this issue since October?

Full error:

Window load time: 761ms 
/Applications/Atom.app/Contents/Resources/app/src/window-bootstrap.js:18
RangeError: Maximum call stack size exceeded
  at DisplayBuffer.module.exports.DisplayBuffer.updateScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:1246:17)
  at DisplayBuffer.module.exports.DisplayBuffer.updateAllScreenLines (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:179:19)
  at new DisplayBuffer (/Applications/Atom.app/Contents/Resources/app/src/display-buffer.js:94:12)
  at new Editor (/Applications/Atom.app/Contents/Resources/app/src/editor.js:76:30)
  at Project.module.exports.Project.buildEditorForBuffer (/Applications/Atom.app/Contents/Resources/app/src/project.js:430:16)
  at /Applications/Atom.app/Contents/Resources/app/src/project.js:192:24
  at _fulfilled (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:787:54)
  at self.promiseDispatch.done (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:816:30)
  at Promise.promise.promiseDispatch (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:749:13)
  at /Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:557:44
  at flush (/Applications/Atom.app/Contents/Resources/app/node_modules/q/q.js:108:17)
  at process._tickCallback (node.js:605:11)

Thanks

[probablycorey]

Opening files larger than 2MB is an edge case that most users don't run into, but I wanted to investigate this.

How to create a large file

Download this file and run it generate-text five-megabytes-of-words.txt 5

Good news

Atom handles 5MB without changing anything.

Bad news

Atom hangs when opening a a file >10MB.

Atom eats up ~300MB or RAM when opening large files.

[probablycorey]

I modified window-bootstrap to test out some memory assumptions. Here is the memory footprint of Atom using a file size of 40MB

Baseline (window-bootstrap.coffee does nothing): 50MB Baseline + file via fs: 96MB Baseline + file via File: 181MB Baseline + TextBuffer: 538MB

[nathansobo]

I'm thinking we probably need to store the TextBuffer data off the v8 heap using a C++ extension, otherwise it hoses GC performance.

[nathansobo]

But there's also a ton of other data such as tokenization etc.

[zcbenz]

Node's Buffer stores data in C++ heap. But if we want to support really big files we have to avoid reading the whole file in memory.

[nathansobo]

Interested in any ideas you have about the design of that @zcbenz. The hardest part might be a bunch of APIs might need to go async to support loading more of the file for certain operations, which could make for more awkward scripting.

[zcbenz]

The basic idea is to only draw the file on needed, but it will need a good caching strategy to make things smooth since drawing file would rely on disk IO, many native apps use memory-mapped files to simplify it, a rough design is:

when opening a large file

1. quickly scans the whole file to count there are how many lines and how long is each line.
2. the editor view computes and draws a virtual scroller bar.
3. the editor view decides which part of file should be read into memory according to current position of scroller.
4. the editor view reads that part of the file and draws them.

when user scrolls the editor view repeat 3 - 4

when the editor view is resized repeat 2 - 4

when the file is changed outside repeat 1 - 4

• step 1 and step4 could be asynchronous.
• the step 1 will need to take the responsibility of tokenization.
• when file is really large that even scanning it takes time we may give up scanning and tokenizing it, the editor view can just show scroller bar with a minimal drag handle, and when user drags the scroller the editor view just roughly computes a start position to read file.

[i4004]

I tried to open ~8gb text file, but nothing just happened :)

[batjko]

I'm not sure how they do it, but Baretail is a pretty impressive log file viewer in that regard. It doesn't care how big your file is and only seems to load portions of it as needed, with very fast seek and search across the entire monster file though, including life refresh of the bottom lines (useful for log files of course) - while keeping the memory footprint quite small.

I assume that's rather difficult to implement in Atom, maybe too much effort for the rare use case?

[nathansobo]

@batjko Thanks for the tip. Long term we definitely want to support files of arbitrary size, but an approach that doesn't load the whole file into memory is going to force us to redesign some of our synchronous APIs to be asynchronous due to the single-threaded nature of JavaScript. We made a pragmatic decision early on to perform editor state manipulations synchronously for a more convenient scripting experience, but we may need to revisit that decision.

[batjko]

@nathansobo Personally, I'm happy with it as it is. I have no use for extreme file sizes in an editor.

I think the huge log file scenario is not really one for editors, but rather for viewers/greppers or whatever you might call that category.

[nathansobo]

Yeah, I pretty much agree. I'm confident we'll get there eventually, but there's a lot of other things we consider more important right now.

[simonzack]

@batjko I don't think this is a rare use case, given the amount of text editors which advertise this as a feature, and the comments so far on this issue. Log viewers and greppers are nice, but it is nicer to view and edit text in an editor. After all isn't that what a text editor is for.

[detly]

The other aspect to this is the failure mode while it isn't supported. If viewing and editing large text files isn't going to work for the time being, it might still be nice to have a better failure mode than locking up and forcing the user to kill Atom.

[nathansobo]

The failure mode is currently an error message displayed in the console for files exceeding 2mb, which should leave Atom in a usable state. Are you talking about large files that are < 2mb?

[detly]

Yes. This is a 1.4M (generated) Python file.

[izuzak]

Yes. This is a 1.4M (generated) Python file.

@detly Can you share the file for which this is happening? Also, does this happen when you run atom in safe mode? Anything else special about the file (e.g. a really long line)?

[izuzak]

Can you share the file for which this is happening?

Actually, I just tried this myself and Atom was slowing down drastically on large files like that. I thought that wasn't happening on files of that size -- I thought I tried that a while ago but I might be wrong. Sorry about that!

[wb-towa]

I can't believe this doesn't seem to have a very high priority given there have been numerous releases of atom and it still can not match the performance of editors from 20 or more years ago.

If you're going to reinvent the wheel it shouldn't be a step backwards.

[daegalus]

Any news or progress on this? I love atom and everything, but I regularly have to open files bigger than 2MB and I keep having to revert to SublimeText to do it.

I still feel this needs a much higher priority and focus. And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

[simonzack]

And if it requires making things asynchronous, its better to do it now than later, as you gain more cruft and features ontop of a certain design, it is much much harder to change later. or a lot messier.

I agree with this, I think that large file support will change lots of the algorithms (e.g. syntax highlighting, line wrapping, column editing, text search) as well, if they aren't built to handle large files from the start. And that can possibly mean interface changes.

[mark-hahn]

Check out view-tail-large-files. It is read-only with no syntax coloring but it opens multi-gigabyte files with no problem. It designed for log files but can open any text file. It is designed for plugins to add features.

[karloluis]

How come this is a problem on powerful computers that have the amount of RAM to retrieve this large kind of file? (32GB RAM PC)

[simonzack]

@karloluis Because the limit should always be the harddrive, not ram. Nobody wants to use up all their ram to open up a log file. What if you want to have 4 vm's running at the same time?

Large file support also has problems other than just opening the file. Syntax highlighting, column editing also have associated algorithm complexity problems.

[wb-towa]

If you're running so many VMs that memory is a concern even with 32gb then you shouldn't be using Atom full stop. It's memory use is quite high for smaller files compared to something like vim or emacs or even sublime I suspect. CPU usage is also higher than it should be for plain text which will affect your multiple VMs.

If we're being honest with ourselves Atom is reinventing the wheel and with numerous regressions.

[karloluis]

@simonzack @toadwarrior This is pretty worrisome, but I feel like there should be a pretty straight forward solution. We'll see what comes of it. On the other hand, given the variety of systems we are running it on it'd be a great feature to be able to set a maximum memory cache (for files) for the program to run on, make it a bit smoother on lower end systems. Obviously there are some few kinks to work on first. Thanks.

[simonzack]

@toadwarrior That isn't what I mean. Memory isn't usually a concern on my system, as long as programs use it reasonably. I'm even ok with firefox using up 1gb. But if atom suddenly uses up 30gb just to open up a large file, and then start hanging at very simple operations, I don't find it acceptable. I thought you supported this issue, and resolving it means that large files should consume much less ram than their file sizes.

[ilanbiala]

Any progress on this? Opening JS sourcemap files pretty much always crashes current Atom editor.

[renkun-ken]

I use Atom version 0.158.0 and I cannot open a 45MB txt file, the file simply does not show anything.

[mark-hahn]

If you only want to read it I suggest my view-tail-large-files package.

On Wed, Dec 17, 2014 at 5:08 PM, Kun Ren notifications@github.com wrote:

I use Atom version 0.158.0 and I cannot open a 45MB txt file, the file simply does not show anything.

— Reply to this email directly or view it on GitHub https://github.com/atom/atom/issues/307#issuecomment-67426464.

[demisx]

+10

Opening < 2MB file with 50K lines has a 13 seconds lag for me. The, it takes another 10 seconds for the syntax highlighting to apply to the entire file. I believe Atom is gaining a slow editor reputation in the community and it makes it harder to compete with others.

Can't even open files larger than 2 MB. This is a major deal for us preventing switch to Atom.

[LevelTranslation]

I want to use Atom to edit books in markdown. Large files might be an edge case for some, but for me, it's my bread and butter. I want to get away from vim, because looking at prose in monospaced fonts is painful, but it continues to be the only thing that will reliably open a whole book.

[ken-okabe]

Can't even open files larger than 2 MB.

Confirmed here, too. Cannot open some webpacked large file.

This is a major deal for us preventing switch to Atom.

I agree. Even I already have switched to Atom for a while now, it's a matter of major quality as an editor.

[mylanconnolly]

I just want to throw in my $0.02, as well. This is a feature that I would really like. At the company I work for we import some very large text files regularly and when something goes wrong I often have to open the file to verify the format hasn't changed, data isn't corrupted, and that particular lines are what I expect.

Sometimes I will have to edit these files by hand, so write support would be beneficial.

I don't expect it to be fast with such a large file (Sublime certainly isn't), but having it at least work would be an improvement over the present situation.

These files are routinely as large as 10-20 MB in size. This is literally the only reason why I have Sublime Text installed still, as for everything else I prefer Atom.

Would be great to see large file support!

[AbdealiLoKo]

I too need this feature, i frequently open sql files and log files

[mark-hahn]

i frequently open sql files and log files

If you only need to read them try out my view-tail-large-files package.

On Thu, Jan 15, 2015 at 2:15 AM, telkomsel666 notifications@github.com wrote:

1

12 Jan 2015 22:49, "Mylan Connolly" notifications@github.com menulis:

I just want to throw in my $0.02, as well. This is a feature that I would really like. At the company I work for we import some very large text files regularly and when something goes wrong I often have to open the file to verify the format hasn't changed, data isn't corrupted, and that particular lines are what I expect.

Sometimes I will have to edit these files by hand, so write support would be beneficial.

I don't expect it to be fast with such a large file (Sublime certainly isn't), but having it at least work would be an improvement over the present situation.

These files are routinely as large as 10-20 MB in size. This is literally the only reason why I have Sublime Text installed still, as for everything else I prefer Atom.

Would be great to see large file support!

— Reply to this email directly or view it on GitHub https://github.com/atom/atom/issues/307#issuecomment-69580080.

— Reply to this email directly or view it on GitHub https://github.com/atom/atom/issues/307#issuecomment-70065070.

[ilanbiala]

@mark-hahn I'm sure many people greatly appreciate the package, but this is really something that has to be part of the core editor. The Atom team can try to integrate and use parts of your package, which may be a good idea, but they shouldn't ignore this issue just because a package exists. Right now, most people prefer Sublime because it is quicker, more stable, and better designed. Atom will likely improve in the latter two as releases are made and more feedback is given, but the first will be tricky due to the program being written in CS/JS.

[blakegarretson]

Not to beat a dead horse, but I'll add another datapoint and lend some support to the "large file" camp. I edit (not just view) files in the 10MB-200MB range every day. This is common in the scientific computing community, specifically fields like FEA and CFD where input files are large and often require hand editing. I would love to use Atom on FEA input files if it could handle them. I wrote a package to support Abaqus (FEA code) syntax in Atom, and then was promptly disappointed to find almost none of my files would open due to the size limitation.

[mark-hahn]

they shouldn't ignore this issue just because a package exists.

Of course not. I was just offering an option you can use now. Also it also offers many features that shouldn't be in core (actually too many features).

P.S. From my experience core has never been shy to add new features because of existing packages.

On Thu, Jan 15, 2015 at 12:35 PM, Blake Garretson notifications@github.com wrote:

Not to beat a dead horse, but I'll add another datapoint and lend some support to the "large file" camp. I edit (not just view) files in the 10MB-200MB range every day. This is common in the scientific computing community, specifically fields like FEA and CFD where input files are large and often require hand editing. I would love to use Atom on FEA input files if it could handle them. I wrote a package to support Abaqus (FEA code) syntax in Atom, and then was promptly disappointed to find almost none of my files would open due to the size limitation.

— Reply to this email directly or view it on GitHub https://github.com/atom/atom/issues/307#issuecomment-70156711.

[demisx]

Atom 1) being slow and 2) not being able to support large files are the 2 main reasons why we can't switch from SublimeText yet.

[simonzack]

Atom 1) being slow and 2) not being able to support large files are the 2 main reasons why we can't switch from SublimeText yet.

This is precisely the same reason I need to use sublime text instead of atom.

[mark-hahn]

Atom is always going to be slower than ST because it uses Node instead of C++. The advantage of using node is that packages can be easily added by anyone who knows web development. The feature set of Atom will grow rapidly and leave the other programming editors in the dust, including ST.

I personally use notepad++ for the occasions I need to quickly open a simple file. I keep an Atom window open all the time for each project I'm developing. Editing a file and program development are quite different endeavors.

If you insist on using one editor for all editing then Atom will probably never be right for you.

On Thu, Jan 15, 2015 at 1:09 PM, simonzack notifications@github.com wrote:

Atom 1) being slow and 2) not being able to support large files are the 2 main reasons why we can't switch from SublimeText yet.

This is precisely the same reason I need to use sublime text instead of atom.

— Reply to this email directly or view it on GitHub https://github.com/atom/atom/issues/307#issuecomment-70161992.

[simonzack]

Atom is always going to be slower than ST because it uses Node instead of C++.

I agree, but it doesn't have to be so slow to the degree that you're constantly aware of it. I think this is achievable. Look at brackets, for example.

[demisx]

Agreed. It can be slower, but not slow. Also, the 2MB file limit is a must fix for any text editor.

[mark-hahn]

Also, the 2MB file limit is a must fix for any text editor.

This is something that can be fixed unlike the speed problem. However it will be a really tough nut to crack. It would require a major refactoring of a large part of Atom. It certainly won't be part of 1.0 but might be in 2.0 or 3.0.

is a must fix for any text editor.

But not for a programming editor. If your source files are over 2MB then you really need to rethink your app's architecture.

[demisx]

@mark-hahn There are concatenated files that exceed 2MB. Has nothing to do with the app's arch.