envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.79k stars 4.76k forks source link

How to set heap memory release rate? #35840

Open anvo1115 opened 1 month ago

anvo1115 commented 1 month ago

Title: How to set heap memory release rate?

Description: Previously I created an issue https://github.com/envoyproxy/envoy/issues/30280. And we waited for https://github.com/envoyproxy/envoy/pull/30353 Now we use envoy v1.30.2. We deploy it on the Kubernetes. We configured overload_manager. MAX_HEAP_SIZE_BYTES is 90 MB + we set 10MB for container docker usage.

overload_manager:
  refresh_interval: 0.25s
  resource_monitors:
    - name: "envoy.resource_monitors.fixed_heap"
      typed_config:
        '@type': type.googleapis.com/envoy.extensions.resource_monitors.fixed_heap.v3.FixedHeapConfig
        max_heap_size_bytes: ${MAX_HEAP_SIZE_BYTES}
  actions:
    - name: "envoy.overload_actions.disable_http_keepalive"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.92
    - name: "envoy.overload_actions.stop_accepting_requests"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.95
    - name: "envoy.overload_actions.stop_accepting_connections"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.98
    - name: "envoy.overload_actions.shrink_heap"
      triggers:
        - name: "envoy.resource_monitors.fixed_heap"
          threshold:
            value: 0.85

And still I don't see that heap memory is released. We send GET request up to ~ 450 requests per second for load testing. I can see that after some time the memory stats gave the following snapshots:

{
 "allocated": "25683496",
 "heap_size": "56623104",
 "pageheap_unmapped": "0",
 "pageheap_free": "4202496",
 "total_thread_cache": "15993768",
 "total_physical_bytes": "61415902"
}

{
 "allocated": "16948136",
 "heap_size": "62914560",
 "pageheap_unmapped": "0",
 "pageheap_free": "5685248",
 "total_thread_cache": "24277664",
 "total_physical_bytes": "67707358"
}

{
 "allocated": "19265368",
 "heap_size": "67108864",
 "pageheap_unmapped": "0",
 "pageheap_free": "5914624",
 "total_thread_cache": "23830904",
 "total_physical_bytes": "71901662"
}

{
 "allocated": "20356456",
 "heap_size": "71303168",
 "pageheap_unmapped": "0",
 "pageheap_free": "5791744",
 "total_thread_cache": "27204976",
 "total_physical_bytes": "76095966"
}

{
 "allocated": "43706392",
 "heap_size": "79691776",
 "pageheap_unmapped": "0",
 "pageheap_free": "5939200",
 "total_thread_cache": "25506720",
 "total_physical_bytes": "84484574"
}

{
 "allocated": "21702552",
 "heap_size": "79691776",
 "pageheap_unmapped": "0",
 "pageheap_free": "10895360",
 "total_thread_cache": "27813728",
 "total_physical_bytes": "84484574"
}
image

Why isn't heap released? probably we need to set some extra properties to release it?

github-actions[bot] commented 4 days ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.