Too high memory usage on history service

[dhiaayachi]

Expected Behavior

Require less memory usage.

Actual Behavior

container resources:

server metrics:

go tool pprof "http://localhost:7936/debug/pprof/heap"

Steps to Reproduce the Problem

1. create one thousand workflows

   temporal-dynamic-config

   apiVersion: v1
   data:
   dynamic_config.yaml: |-
   history.cacheInitialSize:
   - value: "10"
   history.cacheMaxSize:
   - value: "50"
   history.cacheTTL:
   - value: "0.5"
   history.eventsCacheInitialSize:
   - value: "10"
   history.eventsCacheMaxSize:
   - value: "50"
   matching.useOldRouting:
   - value: "false"
   kind: ConfigMap
   metadata:
   annotations:
   meta.helm.sh/release-name: temporal
   meta.helm.sh/release-namespace: temporal
   creationTimestamp: "2022-11-09T04:02:55Z"
   labels:
   app.kubernetes.io/instance: temporal
   app.kubernetes.io/managed-by: Helm
   app.kubernetes.io/name: temporal
   app.kubernetes.io/part-of: temporal
   app.kubernetes.io/version: 1.20.2
   helm.sh/chart: temporal-0.20.2
   name: temporal-dynamic-config
   namespace: temporal
   resourceVersion: "64727569"
   uid: 01925a28-4772-42e9-891e-bdb467bf496f

Specifications

• Version: temporalio/server:1.20.2
• Platform: k8s