Memory leak in Cloud Files directory and file calls?

[plribeiro3000]

I'm using Fog to access Rackspace Cloud Files within EventMachine. I seem to be getting a memory leak related to the Rackspace::Service#request method. At this point I have not managed to separate out the code to make a simple example of the issue, but I'd like to see if the description of my current investigation can trigger any ideas around the real source of the problem. I've tried various Fog versions and have currently landed at 1.27.0 for testing.

I create a Fog connection up front and to get a single directory by name within an HTTP request handled by EventMachine (em-http-server)

service = Fog::Storage.new(req_opt)
# within an EventMachine deferred request
  service.directories.get(@full_container_name)
# end the EventMachine request

I have separated out the Fog stuff, and have just returned data through my server and there is no leak. As soon as I hit service.directories.get the memory usage grows and grows.

Now if I just do:

(0..50).each do |d|
  service.directories.get(@full_container_name)
end

The memory does not grow more than for a single call. So I believe that something is being held onto inside the first call within an EventMachine request.

If I do service.directories.all the memory does appear to leak too in a similar way, so its not just the #get method at fault.

For testing, I reduced the call to just do

service.directories.do_nothing

within the EM deferred request, where I created a dummy method in the Rackspace::Directories class. The memory does not grow in the same way.

Inside the Rackspace::Real class I created a similar dummy method and call this instead of Rackspace::Real#get_container, just to make sure. The memory jumps around a little more but does not grow endlessly.

This suggest to me that there is something happening inside the actual request that is not letting go. So I have just returned from Rackspace::Service#request immediately before @connection.request. Logging the @connection variable shows this connection is always the same (as expected). The memory does not grow if I don't call the @connection.request method.

When the @connection.request is reintroduced the memory grows indefinitely.

I'm assuming at this point I am close to hitting Excon code. But before I start wasting time over there too, I want to see if any of this discussion means anything to anybody here. I'm open to offers!

Original issue opened by @philayres at fog/fog#3442.

[plribeiro3000]

cc/ @philayres @fcheung @geemus @mdarby @gabecase @kert-io

[georgepalmer]

We are also seeing this problem. Memory usage grows indefinitely with every request and we're seeing process hit a few GB in size.

[geemus]

Hmm. It looks like it is creating a new identity service object every time it needs to auth (instead of just holding one reference to an identity service which it would re-use). My guess is that this is causing the bloat in some way as the auth is somewhat short-lived. Not really certain though. Maybe we should cache that reference to avoid re-initializing that service?

[mdarby]

I've run through the scenario and yes, based on your findings I'd say this is somewhere on the Fog::Service side of things? Fog::Storage::Rackspace#request simply calls super in this case.

[geemus]

Yeah, I think maybe it uses load, instead of require for reasons that I forget, but it might well cause the issue. Storing an instance variable of tho connection (or similar) and re-using would probably fix this, but I don't have creds/etc to really test it.

[mdarby]

I ran through three tests of 1000 requests each and have logged the following memory info:

{:lowest=>1.6, :highest=>14.9, :average=>10.153293413173651, :median=>9.9}
{:lowest=>2.1, :highest=>14.7, :average=>10.192015968063867, :median=>10.0}
{:lowest=>1.0, :highest=>19.6, :average=>9.744311377245506, :median=>9.6}

Script: https://gist.github.com/e2fe8fd0fb804dad7ec7fcb12a37e41b

I will try the same script without reusing the @storage service var.

[mdarby]

Without reusing the @storage var:

{:lowest=>0.1, :highest=>14.4, :average=>6.965069860279448, :median=>7.0}
{:lowest=>0.0, :highest=>10.4, :average=>6.5311377245508995, :median=>6.6}
{:lowest=>0.0, :highest=>11.0, :average=>6.6538922155688605, :median=>6.7}

[georgepalmer]

So to clarify - this is worse performing with the change you made? That's how I'm reading those test results unless I'm missing something

[mdarby]

The tests were over fog-rackspace as it stands now; I can't seem to recreate the massive memory gains without introducing EventMachine, etc. Either this issue lies at the heart of Fog::Core or elsewhere. The fog-rackspace code largely just implements the DSL provided by Fog...