Update documentation with with caution over memory leak.

[idyll]

I was having an issue where my sidekiq workers were leaking memory when sending notifications using apnotic.

The source of the leak is indicated in the code below. It's related to the use of a connection pool when sending notifications.

  def perform(token)
    # build the notification outside of the connection block.
    # if you build the notification inside the block memory associated with it is never released.
    notification       = Apnotic::Notification.new(token)
    notification.alert = "Hello from Apnotic!"

    APNOTIC_POOL.with do |connection|
      # making a notification inside the block will leak the object.
      #  notification       = Apnotic::Notification.new(token)
      #  notification.alert = "Hello from Apnotic!"

      response = connection.push(notification)
      raise "Timeout sending a push notification" unless response

      if response.status == '410' ||
        (response.status == '400' && response.body['reason'] == 'BadDeviceToken')
        Device.find_by(token: token).destroy
      end
    end
  end

We send such a large volume of notifications that we'd end up using over 1GB of memory every 12 hours which would eventually run us out of memory.

The above example is the synchronous code, but the same is true with async. You need to build the notification outside of the pool or you leak.

[ostinelli]

Hello @idyll, Before doing anything I'd like to have an understanding of what is happening.

Some time ago @jcs opened an issue (https://github.com/ostinelli/net-http2/issues/7) regarding a potential leak in the underlying Http2 library, but I have seen no follow up. Maybe @igrigorik can comment?

[igrigorik]

I'm not aware of any leaks in http-2 lib, but that doesn't mean they're not there.. A profile would go a long way here.

[idyll]

Since this is a production system, it's not super easy/convenient to profile. But I am totally willing to do it. @igrigorik can you tell me what you'd like me to capture if I profile this. I only want to do it once.

@ostinelli I assure you, there's a leak if you make that object in the connection block. Pulling it out of the block fixes it. I have tested the code both ways - ran in production overnight - with only that change. Sending around 35K notifications per hour.

I do suspect there's likely an unrelated leak in the http-2 lib based on the memory usage that I am seeing.

[ostinelli]

@ostinelli I assure you, there's a leak if you make that object in the connection block. Pulling it out of the block fixes it.

This is hardly a fix, maybe a hack. :)

I may have found the culprit and a solution for this, described in https://github.com/ostinelli/net-http2/issues/7.

[idyll]

I don't think it's related to multi streams. If I force the notifications through one at a time I still get the leak. I think it's related to the connection pool block. (But I could be wrong.)

[idyll]

I will fork the http-2 gem and try the fix in ostinelli/net-http2#7. I am really curious to see what the result of that is.

[danielrhodes]

I believe I'm having a similar issue. Except it looks like fsevent_watch has been spawned a bunch of times. Is apnotic or the http2 library spawning new threads anywhere? This problem appears to go away when I don't use Apnotic.

[danielrhodes]

@ostinelli Any idea on this? I have to remove this gem if I can't find a solution. Even in development it is eating up all available resources in my system.

[ostinelli]

@danielrhodes what kind of idea should I have? I have traced down the memory leak to an external gem and if you follow the references you'll see that it's currently being evaluated and should be released this week. :)

More info here: https://github.com/igrigorik/http-2/issues/73

[ostinelli]

Except it looks like fsevent_watch has been spawned a bunch of times

Not sure I understand what you mean by this. @danielrhodes if you are referring to something else, can you please open another issue and detail what you are experiencing so we can investigate? Thanks.

[ostinelli]

Closed as solved from source.