search

ostinelli / net-http2

NetHttp2 is an HTTP/2 client for Ruby.
MIT License
140 stars 31 forks source link

Memory leak with many async requests on a single client #41

Open jhottenstein opened 4 years ago

jhottenstein commented 4 years ago

I believe I have isolated a memory leak in net-http2 (0.18.2) or http-2 (0.10.1). It appears that the client is hanging onto the requests unless client.close is called.

I followed the format of #7 - run the below code and then use the script at https://blog.skylight.io/hunting-for-leaks-in-ruby/ to generate a diff 

# frozen_string_literal: true

require 'net-http2'
require 'objspace'

ObjectSpace.trace_object_allocations_start

STREAMS = 20
COUNT = 1000 / STREAMS
DUMPS = 3

def make_requests(client, j)
  COUNT.times do |i|
    STREAMS.times do |h|
      puts "#{j}:#{i}:#{h}"

      request = client.prepare_request(:get, "/?#{rand * 1_000_000}")
      request.on(:headers) { |_headers| puts "headers - #{request.object_id}" }
      request.on(:body_chunk) { |_chunk| puts "chunk - #{request.object_id}" }
      request.on(:close) { puts "close - #{request.object_id}" }
      client.call_async(request)
    end
    client.join
    sleep 0.5
  end
  #client.close - uncomment to clear memory leak
end

client = NetHttp2::Client.new("https://http2.akamai.com/")
client.on(:error) { |e| puts 'error' }

DUMPS.times do |j|
  make_requests(client, j)
  file_name = "heap-#{j}.dump"
  file = File.open(file_name, 'w')
  GC.start
  ObjectSpace.dump_all(output: file)
  file.close
end
$ ruby make_requests.rb # without client.close
...
$ ./diff.rb heap-0.dump heap-1.dump heap-2.dump
Leaked 16988 ARRAY objects of size 0/679520 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 8989 DATA objects of size 0/719120 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12
Leaked 6993 IMEMO objects of size 0/279720 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12
Leaked 3998 DATA objects of size 0/303848 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696
Leaked 3998 IMEMO objects of size 0/159920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696
Leaked 3998 DATA objects of size 0/319840 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10
Leaked 2995 ARRAY objects of size 0/119800 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:9
Leaked 2050 IMEMO objects of size 0/82000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10
Leaked 2000 DATA objects of size 0/172000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:687
Leaked 1999 IMEMO objects of size 0/79960 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1999 HASH objects of size 0/1159072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1998 OBJECT objects of size 0/287712 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:674
Leaked 1998 ARRAY objects of size 0/79920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/stream.rb:88
Leaked 1997 DATA objects of size 0/159760 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41
Leaked 1000 OBJECT objects of size 0/112000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:45
Leaked 1000 DATA objects of size 0/64000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:13
Leaked 1000 STRING objects of size 0/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:8
Leaked 1000 OBJECT objects of size 0/104000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:86
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:315
Leaked 1000 STRING objects of size 3000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:28
Leaked 1000 STRING objects of size 20000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:32
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:17
Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:8
Leaked 999 HASH objects of size 0/231768 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:7
Leaked 999 STRING objects of size 19134/39960 at: make_requests.rb:17
Leaked 997 DATA objects of size 0/71784 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:12
Leaked 997 HASH objects of size 0/231304 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:19
Leaked 996 HASH objects of size 0/231072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:22
Leaked 23 ARRAY objects of size 0/920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/compressor.rb:181
Leaked 1 STRING objects of size 27/68 at: make_requests.rb:37

Total Size: 42161/6505068

$ ruby make_requests.rb # without client.close
...
$ ./diff.rb heap-0.dump heap-1.dump heap-2.dump

Leaked 2 STRING objects of size 54/136 at: make_requests.rb:37

Total Size: 54/136