Direct Memory statistics

[leventov]

Since Memory is not ByteBuffers, it's not covered by JMX statistics of direct byte buffer allocation. It's viable to add at least basic atomic counters, like total direct memory allocated, total direct memory mapped. I suggest to expose them as static methods in Memory: Memory.totalDirectMemoryAllocated(), etc.

[leventov]

@leerho FYI

[niketh]

@leventov Very good suggestion 👍

[leerho]

@leventov @niketh @cheddar

Done. I implemented two atomic counters accessible via the Memory class.

  //MONITORING
  /**
   * Gets the current number of active direct memory allocations.
   * @return the current number of active direct memory allocations.
   */
  public static long getCurrentDirectMemoryAllocations() {
    return ResourceState.currentDirectMemoryAllocations_.get();
  }

  /**
   * Gets the current size of active direct memory allocated.
   * @return the current size of active direct memory allocated.
   */
  public static long getCurrentDirectMemoryAllocated() {
    return ResourceState.currentDirectMemoryAllocated_.get();
  }

Someone else can write the MXBean as it doesn't really need to be built into the memory package.

[leventov]

Thanks @leerho. What about also accounting "plain direct" and "file-mapped" separately?

[leerho]

So, can you give me some reasons why this distinction is important?

On Wednesday, March 29, 2017 5:38 PM, Roman Leventov <notifications@github.com> wrote:

Thanks @leerho. What about also accounting "plain direct" and "file-mapped" separately?— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[leventov]

In Druid mapped memory is assumed to be paged out to disk routinely, so seeing high volume of it shouldn't worry us.

Bursts in "plain direct" memory allocations and observable leaks: volume of direct memory grows over instance lifetime and doesn't stabilize at any point - should be investigated. If it is mixed with mapped memory, it's hard to identify bursts and leaks in the first place.

[leerho]

Done.

[leerho]

@leventov

I wrote some simple-mined unit tests. However, if you could actually write an MXBean and hook it to Jconsole, do some plain direct and maps to make sure it works it would be helpful. Then we could contribute that as an example and put it on both Druid and DataSketches websites.