Is there a way to track memory in armeria server?

[person2713]

Hello, I am wondering there is a way to track memory state of armeria server? I found default armeria metrics, something like that:

# HELP armeria_executor_queued_tasks The approximate number of tasks that are queued for execution
# TYPE armeria_executor_queued_tasks gauge
armeria_executor_queued_tasks{name="blockingTaskExecutor",} 0.0
# HELP armeria_executor_queue_remaining_tasks The number of additional elements that this queue can ideally accept without blocking
# TYPE armeria_executor_queue_remaining_tasks gauge
armeria_executor_queue_remaining_tasks{name="blockingTaskExecutor",} 2.147483647E9
# HELP armeria_executor_idle_seconds  
# TYPE armeria_executor_idle_seconds summary
armeria_executor_idle_seconds_count{name="blockingTaskExecutor",} 0.0
armeria_executor_idle_seconds_sum{name="blockingTaskExecutor",} 0.0
# HELP armeria_executor_idle_seconds_max  
# TYPE armeria_executor_idle_seconds_max gauge
armeria_executor_idle_seconds_max{name="blockingTaskExecutor",} 0.0
# HELP armeria_executor_pool_size_threads The current number of threads in the pool
# TYPE armeria_executor_pool_size_threads gauge
armeria_executor_pool_size_threads{name="blockingTaskExecutor",} 0.0 

but it seems that all of them shows unrealistic results

[jrhee17]

there is a way to track memory state of armeria server?

Have you checked out https://armeria.dev/docs/advanced-metrics? You can view metrics such as request count, active requests, response time, etc...

If you had a particular metric that you would like to see, let us know. It might already exist, or we may be able to add more metrics.

it seems that all of them shows unrealistic results

I'm guessing it seems unrealistic in that the all values are 0.

This is most likely due to the fact that blockingTaskExecutor isn't used in your server (blockingTaskExecutor is used for certain services like FileService, or if you specify the blocking option for certain services)

[person2713]

@jrhee17 Is there a way to see the state of queue, that is used for saving incoming requests? I want to build a dashboard where I can see memory footprint of application, I thought that netty (eventLoop) works in following way:

1. save incoming request in queue
2. enentLoop takes first task from the queue and assigned it to worker (worker is a thread from the thread pool)
3. save result in outcoming queue

[jrhee17]

I understood that you would like to visualize the memory footprint of the executor responsible for requests/responses. CommonPools.workerGroup is the default EventLoopGroup that spawns EventLoops (which I think is what you would like to monitor)

I guess it is possible to add metrics using micrometer like such to see basic metrics (I'm yet to look closely what exactly what metrics would be collected correctly since EventLoopGroup is a specific type of ScheduledExecutorService)

ExecutorServiceMetrics.monitor(NoopMeterRegistry.get(), CommonPools.workerGroup(), "worker");

Having said this, this might still not yield the metrics you want - it might help to track the following issue: https://github.com/micrometer-metrics/micrometer/issues/522

[trustin]

You might be interested in tracking JVM memory usage. If that's the case, please use the metric binder implementations from Micrometer: https://github.com/micrometer-metrics/micrometer/tree/main/micrometer-binders/src/main/java/io/micrometer/binder/jvm

[ikhoon]

If you are not familiar with the metrics of Micrometer, you may want to see the following example: https://github.com/line/centraldogma/blob/ed507c70c9e25cfaff4bee25050aca06b12b4843/server/src/main/java/com/linecorp/centraldogma/server/CentralDogma.java#L784-L792

[person2713]

Thank you a lot!

[person2713]

@jrhee17 maybe you know why I have 0 all the time in active_request metric

armeria_server_active_requests{hostname_pattern="*",method="getCreditAgreementSignRules",service="ru.alfabank.thrift.front.service.credit.CreditFrontService$AsyncIface",} 0.0
armeria_server_active_requests{hostname_pattern="*",method="GET",service="SpringWebFlux",} 0.0
armeria_server_active_requests{hostname_pattern="*",method="HEAD",service="SpringWebFlux",} 0.0
armeria_server_active_requests{hostname_pattern="*",method="getPostAuthData",service="ru.alfabank.thrift.front.service.FrontService$AsyncIface",} 0.0
armeria_server_active_requests{hostname_pattern="*",method="requestConfirmationCode",service="ru.alfabank.thrift.front.service.accounts.AccountsFrontService$AsyncIface",} 0.0

[ikhoon]

armeria_server_active_requests is a snapshot of currently running services. If your service ends in a short time, it would not be recorded. https://github.com/line/armeria/blob/c8d1f5ef142f65fd79bd01b87d8155d220c9d0b9/core/src/main/java/com/linecorp/armeria/internal/common/metric/RequestMetricSupport.java#L84-L88

[jrhee17]

As @ikhoon already explained, armeria_server_active_requests represents the number of "active" requests when the snapshot is taken.

This metric is collected via a micrometer gauge which is incremented every time a request is received, and decremented once the response is fully written. Hence, the metric may be very different depending on the timing which you decide to collect the metrics.

jfyi, if you would like to view cumulative request counts, you can use armeria_server_requests

[person2713]

Thank you!