tenderlove
commented
7 years ago @alyssais hi! Thanks for reporting this. I think what we're hitting here is a cache increase. libgit2 keeps a cache of diffs for each call to diff. The life of that cache is the same length as the Rugged::Repository object, so in the case of your sample program the cache life will be the same as the life of the program. The default maximum cache size seems rather large, and unfortunately Rugged doesn't give us any insight in to the cache usage. I've opened #696 so we can see from Ruby land what the cache size / usage is. libgit2 provides API for tuning (or disabling) that cache, but I did't add API for that.
@kivikakk was able to reproduce the leak with a script based on your script, and I've pasted it below:
require "rugged"
def diff_parents(repo, commit)
tbd = [commit]
t = 0
while tbd.length > 0
commit = tbd.shift
t += 1
puts "#{t} #{tbd.length}" if t % 100 == 0
commit.parents.each { |parent| parent.diff(commit); tbd << parent }
tbd = tbd[0..100]
end
end
repo = Rugged::Repository.new("rails")
diff_parents repo, repo.head.targetI found that if I created a new Repository object every once in a while, I could get the memory usage to plateau:
require "rugged"
def diff_parents(repo, commit)
tbd = [commit]
t = 0
while tbd.length > 0
commit = tbd.shift
t += 1
puts "#{t} #{tbd.length}" if t % 100 == 0
commit.parents.each { |parent| parent.diff(commit); tbd << parent }
tbd = tbd[0..100]
if t % 100 == 0 # Limit repo cache size
tbd.map!(&:oid)
repo = Rugged::Repository.new repo.path
tbd.map! { |id| repo.lookup id }
end
end
end
repo = Rugged::Repository.new("rails")
diff_parents repo, repo.head.targetMemory usage should already plateau around this size plus whatever other memory Ruby is using. Do you ever see a plateau? If you rework your program to create new Repository objects does it reduce consumption (I know allocating new repo objects is a huge hack 🙇)?
Thanks in advance!
Note: the reason I think it's a cache issue is we're not seeing any leaked memory reported from leaks or valgrind.
alyssais
arthurschreiber
kivikakk
I'm using Rugged in an application that uses Rugged to generate thousands of diffs in quick succession, and I'm noticing memory growing steadily as it runs.
I've managed to reproduce the problem in this reduced test case, which just generates a lot of diffs with a big repo (rails/rails). You can run this script and watch the memory consumption just go up and up and up.
I don't have the C knowledge to figure out the cause of the leak, so this could be a libgit2 issue rather than a Rugged-specific one.