Exploding memory consumption on large files

[notEvil]

Hi,

I was working on a large file (80k lines) and my PC started to freeze. So I started investigating and noticed that Neovim, when reaching the end of the file, would slow but steadily use up all the memory (many GB) and eventually cause the system freeze. And after quite some time figuring out why that is, I tracked it all the way to this:

diff --git a/lua/vgit/core/ReadStream.lua b/lua/vgit/core/ReadStream.lua
index 181ccfc..dc6b114 100644
--- a/lua/vgit/core/ReadStream.lua
+++ b/lua/vgit/core/ReadStream.lua
@@ -43,14 +43,14 @@ function ReadStream:wrap_callback(callback)
 end

 function ReadStream:start()
-  local stdout_result = ''
+  local stdout_table = {}
   local stderr_result = ''
   local stdout = vim.loop.new_pipe(false)
   local stderr = vim.loop.new_pipe(false)

   local on_stdout = function(_, chunk)
     if chunk then
-      stdout_result = stdout_result .. chunk
+      table.insert(stdout_table, chunk)
     end
   end

@@ -72,7 +72,7 @@ function ReadStream:start()
       stderr:close()
     end

-    self:parse_result(stdout_result, self.spec.on_stdout)
+    self:parse_result(table.concat(stdout_table, ""), self.spec.on_stdout)
     self:parse_result(stderr_result, self.spec.on_stderr)

     if self.spec.on_exit then

The blame takes a lot of time but eventually yields ~27MB. And with the fix, the memory consumption doesn't increase significantly.

I have to note that I use vgit just for the gutter and navigation, and very rarely to preview hunks. There might me other operations with the same issue.

edit: the blame is IO intensive and sshfs slows it down tremendously from ~7s local to ~230s remote (local ethernet). This could increase the number of string concats and might explain why this issue hasn't surfaced earlier.