errors appear after a while, always fixed by restart

jjb commented 7 years ago

My setup (https://github.com/waterlink/rack-reverse-proxy/wiki/How-to-stream-a-WordPress-blog-through-a-Rack-app) will work quite well, and then I'll suddently start to get errors that look like this (i replaced my blog host with blog.example.com):

Errno::ECONNREFUSED: Connection refused - connect(2) for "blog.example.com" port 80
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 904 in initialize
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 904 in open
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 904 in block in connect
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb" line 93 in block in timeout
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb" line 103 in timeout
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 902 in connect
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 887 in do_start
File "/app/vendor/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb" line 882 in start
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-proxy-0.6.0/lib/rack/http_streaming_response.rb" line 70 in session
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-proxy-0.6.0/lib/rack/http_streaming_response.rb" line 59 in response
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-proxy-0.6.0/lib/rack/http_streaming_response.rb" line 29 in headers
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb" line 165 in rack_response_headers
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb" line 156 in build_response_headers
File "/app/vendor/bundle/ruby/2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb" line 152 in response_headers
...

A restart always solves the problem.

So my guess is it's some sort of leak of a resource, such as memory, file descriptors, or something else in the network stack. In theory a memory leak would present as memory problems with my app overall, which doesn't seem to be the case. But I'm not ruling anything out.

Looking through the code, here are candidates for possible resource leak problems:

this call to `clone`

https://github.com/waterlink/rack-reverse-proxy/blob/master/lib/rack_reverse_proxy/rule.rb#L106 -- looks like that url variable is something that can have call called on it, so it's not a string. i attempted to follow the code to figure out what it is, but was unable to.

http connections not being closed?

https://github.com/waterlink/rack-reverse-proxy/blob/master/lib/rack_reverse_proxy/roundtrip.rb#L200-L202 — target_response is a Rack::HttpStreamingResponse, which the documentation says "Wraps the hacked net/http in a Rack way". Is there a resource that needs to be closed? Looking in the code for rack-proxy, I find the http session opened here, but finish is never called https://github.com/ncr/rack-proxy/blob/master/lib/rack/http_streaming_response.rb#L64 -- maybe this is done intentionally, so that the connection will be reused? I don't know what the behavior is if the remote server times out the persistent connection, will net:http close on its side, or perpetually keep resources around?

here's the relevant net::http docs https://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html

If you wish to re-use a connection across multiple HTTP requests without automatically closing it you can use ::new instead of ::start. request will automatically open a connection to the server if one is not currently open. You can manually close the connection with finish.
finish Finishes the HTTP session and closes the TCP connection. Raises IOError if the session has not been started.

jjb commented 7 years ago

/cc @ncr

waterlink commented 7 years ago

@jjb thank you for the report. I have created a minimal rack app with rack-reverse proxy and enabled rack-mini-profiler on it. Here are the results:

Total allocated: 1195411 bytes (1339 objects)
Total retained:  6865 bytes (89 objects)

allocated memory by gem
-----------------------------------
   1105355  ruby-2.4.0/lib
     32850  other
     29469  rack-proxy-0.6.0
     16080  rack-2.0.1
     11577  rack-reverse-proxy-0.11.0
        80  rack-reverse-proxy-bench/app

allocated memory by file
-----------------------------------
   1050072  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb
     32850  <internal:prelude>
     29429  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb
     20985  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb
     18668  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb
     12262  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb
      7968  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb
      6770  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb
      2680  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb
      2596  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb
      2433  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb
      1968  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb
      1858  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb
      1376  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb
      1056  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/middleware.rb
      1040  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb
       920  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/builder.rb
       120  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb
        80  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/common.rb
        80  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/http.rb
        80  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb
        80  /Users/oleksii/workspace/rack-reverse-proxy-bench/app.ru
        40  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/http_streaming_response.rb

allocated memory by location
-----------------------------------
   1049712  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:83
     32850  <internal:prelude>:77
     20057  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190
     18189  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
      8856  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
      5200  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
      3302  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
      2704  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:180
      2682  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
      2480  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:437
      2480  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:454
      2480  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:29
      2400  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:442
      2300  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:55
      2238  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62
      1800  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:14
      1796  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:331
      1680  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:13
      1600  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57
      1440  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:12
      1200  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:15
      1200  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
      1160  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
      1152  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:14
      1129  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:48
      1056  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:22
       984  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:22
       946  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:35
       880  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:980
       808  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:229
       800  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:216
       600  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:28
       576  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:24
       576  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:53
       546  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
       528  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:52
       480  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:31
       480  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:214
       480  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:167
       472  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:955
       456  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/builder.rb:55
       456  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/middleware.rb:21
       440  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:235
       440  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:239
       400  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64
       336  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:635
       320  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:60
       320  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:151
       320  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:41
       288  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/middleware.rb:15

allocated memory by class
-----------------------------------
   1049552  Thread
    103096  String
     16024  MatchData
     10680  Array
      8880  Hash
      2480  Rack::Utils::HeaderHash
      1003  Regexp
       912  Class
       480  URI::HTTP
       336  Net::HTTP
       240  Proc
       240  TCPSocket
       208  RackReverseProxy::Rule::Matches
       160  RackReverseProxy::RoundTrip
       128  Method
       120  Net::HTTP::Get
       120  Net::HTTPNotFound
       120  URI::Generic
        88  Net::BufferedIO
        88  RackReverseProxy::Rule::Candidate
        80  Rack::HttpStreamingResponse
        72  Rack::Builder
        72  RackReverseProxy::ResponseBuilder
        72  RackReverseProxy::Rule
        40  Rack::Request
        40  Rack::URLMap
        40  RackReverseProxy::Middleware
        40  Range

allocated objects by gem
-----------------------------------
       593  ruby-2.4.0/lib
       367  rack-proxy-0.6.0
       198  rack-2.0.1
       177  rack-reverse-proxy-0.11.0
         2  other
         2  rack-reverse-proxy-bench/app

allocated objects by file
-----------------------------------
       366  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb
       291  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb
       151  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb
       144  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb
       141  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb
        41  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb
        32  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb
        28  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb
        25  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb
        22  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb
        14  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb
        10  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb
         8  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/builder.rb
         7  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/middleware.rb
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/common.rb
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/http.rb
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb
         2  /Users/oleksii/workspace/rack-reverse-proxy-bench/app.ru
         2  <internal:prelude>
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/http_streaming_response.rb

allocated objects by location
-----------------------------------
       234  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
       135  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
        98  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
        71  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
        60  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
        52  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62
        45  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:55
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57
        36  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190
        33  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:14
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:454
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:15
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:29
        30  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
        29  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
        26  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:331
        26  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:180
        20  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:216
        15  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:22
        13  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
        11  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:28
        10  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64
        10  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:980
         9  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:35
         7  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:48
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:13
         6  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:45
         5  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:235
         5  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:239
         5  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:437
         5  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:442
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:436
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:465
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:37
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:71
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/timeout.rb:83
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:520
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:547
         4  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:22
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:476
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:12
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:24
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:156
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:955
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:14
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:35
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:1480
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:333
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/builder.rb:147

allocated objects by class
-----------------------------------
       977  String
       240  Array
        55  MatchData
        30  Hash
         5  Rack::Utils::HeaderHash
         4  URI::HTTP
         3  Proc
         2  Class
         2  Method
         2  RackReverseProxy::Rule::Matches
         2  Regexp
         1  Net::BufferedIO
         1  Net::HTTP
         1  Net::HTTP::Get
         1  Net::HTTPNotFound
         1  Rack::Builder
         1  Rack::HttpStreamingResponse
         1  Rack::Request
         1  Rack::URLMap
         1  RackReverseProxy::Middleware
         1  RackReverseProxy::ResponseBuilder
         1  RackReverseProxy::RoundTrip
         1  RackReverseProxy::Rule
         1  RackReverseProxy::Rule::Candidate
         1  Range
         1  TCPSocket
         1  Thread
         1  URI::Generic

retained memory by gem
-----------------------------------
      6665  ruby-2.4.0/lib
       200  rack-reverse-proxy-0.11.0

retained memory by file
-----------------------------------
      3642  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb
      1353  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb
       886  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb
       664  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb
       120  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb
        80  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb

retained memory by location
-----------------------------------
      1802  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
      1313  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190
       960  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:14
       800  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
       646  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62
       336  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:635
       240  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:904
       120  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:79
       120  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:30
       120  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43
        88  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:955
        80  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb:51
        80  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:288
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:27
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:87
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:41
        40  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:171

retained memory by class
-----------------------------------
      3961  String
       960  Array
       960  Hash
       336  Net::HTTP
       240  TCPSocket
       120  Net::HTTP::Get
       120  Net::HTTPNotFound
        88  Net::BufferedIO
        80  Rack::HttpStreamingResponse

retained objects by gem
-----------------------------------
        87  ruby-2.4.0/lib
         2  rack-reverse-proxy-0.11.0

retained objects by file
-----------------------------------
        62  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb
        17  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb

retained objects by location
-----------------------------------
        38  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
        20  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
        13  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:14
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:288
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb:51
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:79
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:635
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:904
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:955
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:27
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:30
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:87
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:41
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:171

retained objects by class
-----------------------------------
        57  String
        24  Array
         2  Hash
         1  Net::BufferedIO
         1  Net::HTTP
         1  Net::HTTP::Get
         1  Net::HTTPNotFound
         1  Rack::HttpStreamingResponse
         1  TCPSocket

Allocated String Report
-----------------------------------
        66  ""
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
        12  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:55
        12  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:22
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:980
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:35
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:235
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:34
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:65
         1  <internal:prelude>:77

        63  "-"
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
        20  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
        13  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

        32  "\n"
        11  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:55
        10  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:454
        10  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:29
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:164

        30  "_"
        30  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

        17  ", "
        13  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:180
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:22
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:36

        13  "accept"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:45
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:71
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:35

        11  "cache-control"
         5  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64

        11  "host"
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:1503
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:35

        10  "\t"
        10  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57

        10  " "
        10  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57

        10  "/test"
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:404
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:60
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:62
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:150
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:48
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:1047
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:980
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/http.rb:105
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:59
         1  /Users/oleksii/workspace/rack-reverse-proxy-bench/app.ru:11

        10  "HOST"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:84

        10  "X"
         6  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57

        10  "user-agent"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:45
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:71
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:35

         9  "/"
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:30
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:63
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:44
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:34
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/urlmap.rb:35
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:980

         9  "ACCEPT"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472

         9  "ACCEPT-ENCODING"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "ACCEPT-LANGUAGE"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "ACCEPT_ENCODING"
         9  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "ACCEPT_LANGUAGE"
         9  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "Accept"
         5  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:331
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:45

         9  "CACHE-CONTROL"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "CACHE_CONTROL"
         9  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "CONNECTION"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472

         9  "COOKIE"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472

         9  "USER-AGENT"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "USER_AGENT"
         9  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         9  "VERSION"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:472

         9  "accept-encoding"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:71
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:476
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158

         9  "x-forwarded-for"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:465
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158

         7  "connection"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         7  "cookie"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         7  "http"
         3  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:333
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/generic.rb:1480
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:24

         7  "version"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         6  "::1"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:22
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:180

         6  "C"
         6  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:57

         6  "Cache"
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         6  "Content"
         4  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         6  "Forwarded"
         6  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187

         6  "Host"
         5  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:331

         6  "UPGRADE-INSECURE-REQUESTS"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         6  "UPGRADE_INSECURE_REQUESTS"
         6  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-proxy-0.6.0/lib/rack/proxy.rb:37

         6  "accept-language"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158

         6  "date"
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64

         6  "expires"
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64

         6  "server"
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64

         6  "upgrade-insecure-requests"
         3  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:158

         6  "vary"
         2  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/utils.rb:469
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:64

         5  "/test?pp=profile-memory"
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-2.0.1/lib/rack/request.rb:408
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/rule.rb:151
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:27
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/http.rb:105
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:52

         5  "Cache-Control"
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:215
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:216
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:331
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:187
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

Retained String Report
-----------------------------------
         2  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8"
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         2  "cache-control"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         2  "example.org"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/uri/rfc3986_parser.rb:41

         2  "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
         2  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "\r\n<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" conte"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:190

         1  "/test?pp=profile-memory"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:27

         1  "0"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:288

         1  "1"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "1.1"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43

         1  "1270"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "404"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43

         1  "404-HIT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "9292"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "::1"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/protocol.rb:87

         1  "Accept-Encoding"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "Date: Wed, 29 Mar 2017 19:10:40 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "ECS (iad/18EF)"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "Expires: Wed, 05 Apr 2017 19:10:40 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "HTTP/1.1"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "Last-Modified: Sun, 26 Mar 2017 13:13:23 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "Not Found"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:43

         1  "Sun, 26 Mar 2017 13:13:23 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "Wed, 05 Apr 2017 19:10:40 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "Wed, 29 Mar 2017 19:10:40 GMT"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "__profilin=p%3Dt"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "accept"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "accept-language"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "accept-ranges"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "bytes"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "connection"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "content-length"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "content-type"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "cookie"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "date"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "en-us"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "expires"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "host"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "keep-alive"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "last-modified"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "localhost"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "max-age=0"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "max-age=604800"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "server"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "text/html"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/response.rb:62

         1  "upgrade-insecure-requests"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "user-agent"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "vary"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

         1  "version"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:21

         1  "x-cache"
         1  /Users/oleksii/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/net/http/header.rb:67

jjb commented 7 years ago

science! 🔬

I'm not familiar with reading the output of that tool -- are you able to draw conclusions, or is this just initial data to consider?

waterlink commented 7 years ago

Most interesting parts that I grasp from it are:

retained objects by location
-----------------------------------
         ...
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/response_builder.rb:51
         1  /Users/oleksii/.rvm/gems/ruby-2.4.0/gems/rack-reverse-proxy-0.11.0/lib/rack_reverse_proxy/roundtrip.rb:79
        ....

jjb commented 7 years ago

okay so this code (in master, haven't double checked to make sure 2.4 is the same)

    def target_response
      @_target_response ||= Rack::HttpStreamingResponse.new( #51
        target_request,
        uri.host,
        uri.port
      )
    end

    def build_target_request
      Net::HTTP.const_get(
        source_request.request_method.capitalize
      ).new(uri.request_uri) #79
    end

jjb commented 7 years ago

@waterlink maybe in your test setup you can put in some debug code in target_response to make sure it memoizes once per request and not once per app instantiation? (assuming that's the intention)

waterlink commented 7 years ago

It seems that tool, also, outputs retained strings and objects. All retained strings look like headers and their values. So I'm actually assuming, that most of retained things are the ones that go out of the app through the http response eventually.

waterlink commented 7 years ago

@jjb also, how much load on average does your website/app has, especially the endpoints that proxy to the blog?

jjb commented 7 years ago

how much load on average does your website/app has, especially the endpoints that proxy to the blog?

pretty low, we haven't launched it yet. so we are just working on it internally while we set it up.

i'm doing a test right now to see if file descriptors are being leaked (cat /proc/sys/fs/file-nr on linux)

waterlink commented 7 years ago

@jjb Also, I've just ran GC profiling tool + Apache Bench benchmark in parallel with high concurrency until it started failing and the object counts spiked up, but in a few moments after AB finished all the related object counts has gone down to 1 (the ones that are created for current request). So it seems like unclosed resource so far, not a memory leak.

jjb commented 7 years ago

doing a local test, while stress testing with ab, the number on the left of cat /proc/sys/fs/file-nr went up initially and then leveled, so there's nothing there.

@waterlink have you heard of people having similar problems? i couldn't find any other tickets about this, so I actually figured maybe my setup (https://github.com/waterlink/rack-reverse-proxy/wiki/How-to-stream-a-WordPress-blog-through-a-Rack-app) was at least part of the problem.

jjb commented 7 years ago

with this config.ru

require_relative 'config/environment'

class DebugMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    unclosed_files = 0
    ObjectSpace.each_object(IO) do |f|
      unclosed_files +=1 unless f.closed?
    end
    p "unclosed files: #{unclosed_files}"
    @app.call(env)
  end
end

use DebugMiddleware
run Rails.application

and stress testing with ab, the count will go up to around 60 or 70, then get cleaned up and jump down to 15, climb back up to 60 or 70.

i also tried it with a simpler, synchronous stress test watch --interval 0.1 curl -s --head http://..., with similar results

so what's interesting is that they aren't cleaned up per-request, they are cleaned up by some other system, presumably ruby. so maybe if a indexing bot comes to visit, the barrage of requests won't clean up in time.

I'll try to change the rack-proxy code to explicitly close connections and see if that helps.

waterlink commented 7 years ago

Yeah, I see what you mean. They are cleaned up by ruby's garbage collector. It is definitely an amazing idea to close them at the end of the request. Thank you for the investigation!

jjb commented 7 years ago

okay here we go. with these two debug lines in rack-proxy, "begin_request_hacked" is printed with each request, and "end_request_hacked" is /never/ printed

not sure if a bug in rack-proxy or in how rack-reverse-proxy is using it

/cc @ncr

jjb commented 7 years ago

end_request_hacked is only called when each is called on HttpStreamingResponse.

but only body is called on it in rack-reverse-proxy: https://github.com/waterlink/rack-reverse-proxy/blob/master/lib/rack_reverse_proxy/roundtrip.rb#L200-L202

HttpStreamingResponse#body returns self, so presumably some other code down the line calls something else on it. my first guess was to_s, but HttpStreamingResponse#to_s calls each, which should then call end_request_hacked.

i'll put in some more debug code

jjb commented 7 years ago

only body is called. each and to_s are never called. my guess was read, but putting this into HttpStreamingResponse

    def read
      p ">>> read"
      super
    end

yielded no output.

now to put in some funky metaprogramming code to see what methods are called after it's passed as the rack response...

waterlink commented 7 years ago

@jjb When a rack middleware returns a triple of form [status, headers, body], rack itself will read the body using body.each method. Or at least, if you put some object there that don't have each defined ruby will raise NoMethodError with appropriate message telling that no method each defined on this object.

jjb commented 7 years ago

🤦‍♂️ i was testing with HEAD requests so that's why the body wasn't being read. starting debugging again now with GET

@waterlink thanks for the info re: rack, that's what i assumed but behavior didn't show that... until i switched to GET

jjb commented 7 years ago

each and end_request_hacked are in fact being called, now to figure out if/why end_request_hacked isn't sufficient

jjb commented 7 years ago

BINGO 💥

with this new line in this file of rack-proxy, the unclosed file count remains constant

    def each(&block)
      response.read_body(&block)
    ensure
      session.end_request_hacked
      session.finish # new line
    end

rack-proxy does call end_transport, which calls close on the socket, so I'm not sure about semantic differences or best practices here. but finish is what the documentation for Net::HTTP says to use when not using block style (https://docs.ruby-lang.org/en/2.2.0/Net/HTTP.html#class-Net::HTTP-label-How+to+use+Net-3A-3AHTTP), so seems me that should probably be somewhere in the rack-proxu code.

@ncr, would be great to have your thoughts at this point.

meanwhile i'll experiment if there is a way to get the fix into rack-reverse-proxy

waterlink commented 7 years ago

@jjb Sounds amazing! Thank you for finding this out!

jjb commented 7 years ago

okay, this code in rack-reverse-proxy, with an unmodified rack-proxy, also fixed the problem

    def rack_response
      sio = StringIO.new(target_response.to_s)
      target_response.send(:session).finish
      [target_response.status, response_headers, sio]
    end

options going forward:

fix rack-proxy https://github.com/ncr/rack-proxy/issues/62
monkeypatch rack-proxy
more efficient version of the above code - a wrapper class that reads lines from the target_response one by one and then closed it when done, instead of having to instantiate a whole new object with all the content ahead of time

waterlink commented 7 years ago

The proposed version of def rack_response would work for proxying html pages. It would not work very well with real streaming responses, such as high resolution images, video stream, or file download. So I would say: either fix it in rack-proxy or provide an efficient wrapper.

If you know that your app is not subject to aforementioned problems, you could monkey-patch either rack-proxy or rack-reverse-proxy. Bear in mind, that it would be wise to monkey-patch only at the public API level, not at the internal level. This is because at internal level any semantic-version-compliant refactoring within minor and patch level version bump may break your monkey patch.

jjb commented 7 years ago

I see. I actually don't know if this "leak" of open files is the cause of my problems, or if it's a leak at all. In my test it gets cleaned up regularly, but it seemed to be a potential place that maybe some out of the ordinary circumstance would cause problems to eventually happen where it was a real leak.

From your above comment it seems you know what the semantics of end_transport?

waterlink commented 7 years ago

@jjb I am not sure what is the difference between end_transport and finish. Could be that end_transport only closes the connection, but does not discard some timeouts or even threads that were created for connection (I do see that count of Timeout objects grows when benching, and they never are deleted); finish might do just that.

I guess the only true way to find it out is to look at the ruby stdlib's source code.

jjb commented 7 years ago

🎶 hello Timeout my old friend 🎶

jjb commented 7 years ago

If you don't know the difference, why are you confident that finish will disrupt streaming responses but end_transport will not (IIUC)?

waterlink commented 7 years ago

I wasn't talking about difference of finish and end_transport, rather about difference where we can call them.

Calling them in the context of def rack_response function will read the whole response body into the memory first and only then give it back to the user. It does not actually disrupt anything, it just makes it impossible (or very memory-hungry) to do these streaming responses.

If we call finish and/or end_transport in each function of the response body, it is different deal, because Rack will take bits of response body and give them to the end user/client one at a time (streaming).

waterlink commented 7 years ago

Last 2 paragraphs are too abstract. Here is the example code:

# BAD because it reads the whole body into the memory before giving it to the rack
def rack_response
  sio = StringIO.new(target_response.to_s)
  target_response.send(:session).finish      # or .end_transport
  [target_response.status, response_headers, sio]
end

# GOOD because it will let rack use `#each` and read body piece by piece
def rack_response
  finishing_wrapper = FinishingWrapper.new(target_response)
  [target_response.status, response_headers, finishing_wrapper]
end

class FinishingWrapper
  def initialize(delegatee) @delegatee = delegates; end

  def each(&blk)
    @delegatee.each(&blk)
    @delegatee.send(:session).finish
  end
end

jjb commented 7 years ago

ahh, gotcha.

Calling them in the context of def rack_response function will read the whole response body into the memory first and only then give it back to the user. It does not actually disrupt anything, it just makes it impossible (or very memory-hungry) to do these streaming responses.

To my understanding it is never impossible, it is always memory-hungry. The rack middleware will just gobble up the whole response before sending it on down the line. Assuming that the largest response encountered is acceptable, then this solution is acceptable (if it's solving something...).

But anyway that is perhaps moot because we have a better solution...

class FinishingWrapper
...

😍

Isn't that semantically equivalent to my change to rack-proxy in https://github.com/waterlink/rack-reverse-proxy/issues/41#issuecomment-290236733 ?

waterlink commented 7 years ago

It is. It is just in different library and suffers from probable breaking changes because it uses undocumented API (sending to private method) On Thu, 30 Mar 2017 at 11:28 PM, John Bachir notifications@github.com wrote:

ahh, gotcha.

Calling them in the context of def rack_response function will read the whole response body into the memory first and only then give it back to the user. It does not actually disrupt anything, it just makes it impossible (or very memory-hungry) to do these streaming responses.

To my understanding it is never impossible, it is always memory-hungry. The rack middleware will just gobble up the whole response before sending it on down the line. Assuming that the largest response encountered is acceptable, then this solution is acceptable (if it's solving something...).

But anyway that is perhaps moot because we have a better solution...

class FinishingWrapper ...

😍

Isn't that semantically equivalent to my change to rack-proxy in #41 (comment) https://github.com/waterlink/rack-reverse-proxy/issues/41#issuecomment-290236733 ?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/waterlink/rack-reverse-proxy/issues/41#issuecomment-290550330, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlLFhtvFpzFSyVjWSkMRuXyjET4ghS2ks5rrB55gaJpZM4MordU .

jjb commented 7 years ago

gotcha

yeah when i monkeypatch i do something like this:

raise "need to check if monkeypatch is still correct" unless "1.2.3" == Rack::Proxy::Version

waterlink commented 7 years ago

That is a fantastic idea 💡 On Fri, 31 Mar 2017 at 8:15 PM, John Bachir notifications@github.com wrote:

gotcha

yeah when i monkeypatch i do something like this:

raise "need to check if monkeypatch is still correct" unless "1.2.3" == Rack::Proxy::Version

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/waterlink/rack-reverse-proxy/issues/41#issuecomment-290788067, or mute the thread https://github.com/notifications/unsubscribe-auth/AAlLFgRFqcxBVeSh2j8OShtq7HvfvjcQks5rrUKjgaJpZM4MordU .

ncr commented 7 years ago

Hi there, just pushed a new rack-proxy gem (0.6.1) with a fix from @jjb.

jjb commented 7 years ago

Thanks @ncr!

Closing this now.

jjb commented 7 years ago

Possibly related: https://github.com/puma/puma/pull/1206

waterlink / rack-reverse-proxy