search

eclipse-ee4j / jersey

Eclipse Jersey Project - Read our Wiki:
https://github.com/eclipse-ee4j/jersey/wiki
Other
692 stars 354 forks source link

Revisit the Monitoring feature #4152

Open jansupol opened 5 years ago

jansupol commented 5 years ago
jbescos commented 3 years ago

There are many residual AggregatedValueObject instances that are not trim. These objects are holded by AggregatedSlidingWindowTimeReservoir.

Adding some log lines it is possible to see the issue. In the lines we can see many instances of AggregatedSlidingWindowTimeReservoir.

Something is not working properly because there are PUTs of AggregatedValueObject in many AggregatedSlidingWindowTimeReservoir, but it only trims in one. This causes residual AggregatedValueObject instances that can be seen when you run a heapdump.


New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@600e965c
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5d49099
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6f554bf8
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5cf23ef0
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@4973df1f
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6802def0
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@2d4d5283
PUT: 6799281429504000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281429504000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281429504000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5d49099
PUT: 6799281429504000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6f554bf8
PUT: 6799281429760000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5cf23ef0
PUT: 6799281429760000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@4973df1f
PUT: 6799281429760000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6802def0
PUT: 6799281429760000000. Size: 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@2d4d5283
PUT: 6799281429504000002. Size: 3 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281429504000003. Size: 4 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281429504000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5d49099
PUT: 6799281429504000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6f554bf8
PUT: 6799281429760000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@5cf23ef0
PUT: 6799281429760000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@4973df1f
PUT: 6799281429760000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6802def0
PUT: 6799281429760000001. Size: 2 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@2d4d5283
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@6329d242
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@125efdc4
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@14cbdfc
New instance of org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@338adcf1
PUT: 6799281429504000004. Size: 5 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281685504000000. Size: 6 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281685504000001. Size: 7 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281685504000002. Size: 8 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
PUT: 6799281941504000000. Size: 9 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
TRIM: 6799285342357864448, Cleaned: 5. Remaining 4 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
TRIM: 6799285598869864448, Cleaned: 3. Remaining 1 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f
TRIM: 6799285854869864448, Cleaned: 1. Remaining 0 in org.glassfish.jersey.server.internal.monitoring.AggregatedSlidingWindowTimeReservoir@35a4060f

AggregatedSlidingWindowTimeReservoir@35a4060f is fully cleaned, but the other instances still contains AggregatedValueObject.

Edit: It seems it removes them, but there are 2 constants making it to happen less frequently, so maybe we can make them configurable. We can find this in AbstractSlidingWindowTimeReservoir.

In the constructor: this.window = windowUnit.toNanos(window) << COLLISION_BUFFER_POWER; That makes the window very big. For example if the value windowUnit.toNanos(window) = 1000000000L, then windowUnit.toNanos(window) << COLLISION_BUFFER_POWER = 256000000000L. This window is used to trim AggregatedValueObject that are old enough to be trimmed.

And additionally, there is:

    @Override
    public void update(V value, long time, TimeUnit timeUnit) {
        if (updateCount.incrementAndGet() % ReservoirConstants.TRIM_THRESHOLD == 0) {
            trim();
        }

That TRIM_THRESHOLD is 256, so it only trims once AggregatedSlidingWindowTimeReservoir contains 256 AggregatedValueObject.

The number of instances of AggregatedSlidingWindowTimeReservoir is something that I still need to investigate.