Frontend Service - goroutine (CPU & Memory) Leak

dhiaayachi commented 2 weeks ago

Expected Behavior

There should be no memory leak resulting from objects not being properly garbage collected.

Actual Behavior

Number of objects on Heap keeps growing. This seems to be result in slow increase of cpu & memory usage eventually resulting in outage.

Steps to Reproduce the Problem

Start server

temporal server start-dev --port 7233 --ui-port 8233 --metrics-port 9233

Do not start workflows or make any grpc calls, use the sdks or the web ui. aka keep the usage to a minimum.
Periodically check pprof & metrics. This shows that the goroutinue counts, objects on heap (aka memory allocations for objects) & memory allocations keeps growing perpetually.

runtime.MemStats from from http://localhost:/debug/pprof/heap?debug=1


# runtime.MemStats
# Alloc = 65091800
# TotalAlloc = 318383929504
# Sys = 119521560
# Lookups = 0
# Mallocs = 1641446565
# Frees = 1641167975
# HeapAlloc = 65091800
# HeapSys = 82378752
# HeapIdle = 11141120
# HeapInuse = 71237632
# HeapReleased = 4464640
# HeapObjects = 278590
# Stack = 18284544 / 18284544
# MSpan = 825408 / 960048
# MCache = 19200 / 31200
# BuckHashSys = 2638467
# GCSys = 12561120
# OtherSys = 2667429
# NextGC = 69589984
# LastGC = 1721688648441657922
...
# NumGC = 10489
# NumForcedGC = 0
# GCCPUFraction = 4.835076125703521e-05
# DebugGC = false
# MaxRSS = 178909184



- Flame graph

![flame_graph](https://github.com/user-attachments/assets/44044c3b-2fae-4160-8b87-28f2e6b80407)

- This was observed across local & shared & production environments. Please see the prometheus chart in a production environment where the num_goroutine count kept increasing until a restart. Notice that the "leap" appears to be isolated to the frontend service. The rest seem fine. The CPU usage & memory usage charts followed the same pattern.

![image](https://github.com/user-attachments/assets/034b8b5e-96b2-4879-85ac-18def393131e)

## Specifications
This was observed across multiple versions
  - Server Version: 1.22.5, 1.22.7, 1.23.1, etc.
  - Platform: Linux
  - MTLS enabled
  - Auth disabled

## Links
This issue is potentially related to https://community.temporal.io/t/high-cpu-usage-memory-leakage-on-frontend-service/4246/1

dhiaayachi commented 1 day ago

Temporal Server Memory Leak Issue

This GitHub issue describes a potential memory leak in the Temporal Server, causing a gradual increase in CPU and memory usage, ultimately leading to outages.

Expected Behavior:

Objects should be properly garbage collected, preventing memory leaks.

Actual Behavior:

The number of objects on the heap continues to grow, resulting in increased CPU and memory usage. This ultimately leads to an outage.

Steps to Reproduce:

Start the Temporal Server in dev mode:

temporal server start-dev --port 7233 --ui-port 8233 --metrics-port 9233

Avoid starting any workflows or making any gRPC calls using the SDKs or the Web UI.
Monitor pprof and metrics periodically. This shows a perpetual increase in goroutine counts, objects on the heap (memory allocations for objects), and total memory allocations.

Observations:

pprof heap: The image shows a growing heap, indicating memory allocation without deallocation.
runtime.MemStats: The provided MemStats output reveals a consistent increase in memory allocation (TotalAlloc) and the number of objects on the heap (HeapObjects).
Flame graph: The flame graph shows the Frontend service as the main contributor to the growing heap.
Prometheus chart: The chart confirms a consistent increase in goroutine counts, primarily in the Frontend service.

Specifications:

Server Version: 1.22.5, 1.22.7, 1.23.1, etc.
Platform: Linux
MTLS Enabled: Yes
Auth Disabled: Yes

Potential Related Issues:

High CPU Usage & Memory Leakage on Frontend Service

Recommendations:

Further Investigation: This issue requires further investigation. Consider profiling the Frontend service to identify the specific objects causing the leak.
Temporal Community Forum: Reach out to the Temporal community forum for support and assistance in troubleshooting this issue.
Upgrade Temporal Server: Ensure that you are using the latest stable version of the Temporal Server.
Enable Logging: Increase the logging level in the Frontend service to gather more detailed information about the memory usage.
Review Frontend Service Configuration: Verify that the dynamic configuration values for the Frontend service are appropriately set for your workload.
Check for Third-Party Library Issues: If using any third-party libraries, check for known memory leak issues.

By following these steps, you can hopefully pinpoint the root cause of the memory leak and take corrective actions to ensure a stable and reliable Temporal Service.

dhiaayachi commented 22 hours ago

Thanks for reporting this issue. Could you please tell me what version of Temporal you are using? Is it Temporal Cloud or self-hosted? Also, could you provide the details of your database setup? This will help us diagnose the issue further.

dhiaayachi commented 6 hours ago

Thank you for reporting this issue.

Could you please confirm if the Frontend service was started with the --ui-port flag?

This flag is necessary to enable the UI, which provides insights into the system.

Also, please let me know if you've attempted any troubleshooting steps or tried running the Temporal Server in a different environment?

The Troubleshooting section in our documentation might have some helpful information.

dhiaayachi / temporal