Open jhottenstein opened 4 years ago
I believe I have isolated a memory leak in net-http2 (0.18.2) or http-2 (0.10.1). It appears that the client is hanging onto the requests unless client.close is called.
I followed the format of #7 - run the below code and then use the script at https://blog.skylight.io/hunting-for-leaks-in-ruby/ to generate a diff
# frozen_string_literal: true require 'net-http2' require 'objspace' ObjectSpace.trace_object_allocations_start STREAMS = 20 COUNT = 1000 / STREAMS DUMPS = 3 def make_requests(client, j) COUNT.times do |i| STREAMS.times do |h| puts "#{j}:#{i}:#{h}" request = client.prepare_request(:get, "/?#{rand * 1_000_000}") request.on(:headers) { |_headers| puts "headers - #{request.object_id}" } request.on(:body_chunk) { |_chunk| puts "chunk - #{request.object_id}" } request.on(:close) { puts "close - #{request.object_id}" } client.call_async(request) end client.join sleep 0.5 end #client.close - uncomment to clear memory leak end client = NetHttp2::Client.new("https://http2.akamai.com/") client.on(:error) { |e| puts 'error' } DUMPS.times do |j| make_requests(client, j) file_name = "heap-#{j}.dump" file = File.open(file_name, 'w') GC.start ObjectSpace.dump_all(output: file) file.close end
$ ruby make_requests.rb # without client.close ... $ ./diff.rb heap-0.dump heap-1.dump heap-2.dump Leaked 16988 ARRAY objects of size 0/679520 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41 Leaked 8989 DATA objects of size 0/719120 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12 Leaked 6993 IMEMO objects of size 0/279720 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:12 Leaked 3998 DATA objects of size 0/303848 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696 Leaked 3998 IMEMO objects of size 0/159920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:696 Leaked 3998 DATA objects of size 0/319840 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10 Leaked 2995 ARRAY objects of size 0/119800 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:9 Leaked 2050 IMEMO objects of size 0/82000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:10 Leaked 2000 DATA objects of size 0/172000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:687 Leaked 1999 IMEMO objects of size 0/79960 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41 Leaked 1999 HASH objects of size 0/1159072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41 Leaked 1998 OBJECT objects of size 0/287712 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:674 Leaked 1998 ARRAY objects of size 0/79920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/stream.rb:88 Leaked 1997 DATA objects of size 0/159760 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/emitter.rb:41 Leaked 1000 OBJECT objects of size 0/112000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:45 Leaked 1000 DATA objects of size 0/64000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:13 Leaked 1000 STRING objects of size 0/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:8 Leaked 1000 OBJECT objects of size 0/104000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/client.rb:86 Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/connection.rb:315 Leaked 1000 STRING objects of size 3000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:28 Leaked 1000 STRING objects of size 20000/40000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:32 Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:17 Leaked 1000 HASH objects of size 0/232000 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/callbacks.rb:8 Leaked 999 HASH objects of size 0/231768 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:7 Leaked 999 STRING objects of size 19134/39960 at: make_requests.rb:17 Leaked 997 DATA objects of size 0/71784 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/stream.rb:12 Leaked 997 HASH objects of size 0/231304 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:19 Leaked 996 HASH objects of size 0/231072 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/net-http2-0.18.2/lib/net-http2/request.rb:22 Leaked 23 ARRAY objects of size 0/920 at: /Users/jhottenstein/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/http-2-0.10.1/lib/http/2/compressor.rb:181 Leaked 1 STRING objects of size 27/68 at: make_requests.rb:37 Total Size: 42161/6505068 $ ruby make_requests.rb # without client.close ... $ ./diff.rb heap-0.dump heap-1.dump heap-2.dump Leaked 2 STRING objects of size 54/136 at: make_requests.rb:37 Total Size: 54/136
I believe I have isolated a memory leak in net-http2 (0.18.2) or http-2 (0.10.1). It appears that the client is hanging onto the requests unless client.close is called.
I followed the format of #7 - run the below code and then use the script at https://blog.skylight.io/hunting-for-leaks-in-ruby/ to generate a diff