search

OpenMPDK / SMDK

SMDK, Scalable Memory Development Kit, is developed for Samsung CXL(Compute Express Link) Memory Expander to enable full-stack Software-Defined Memory system
271 stars 60 forks source link

Query regarding Zone configuration #9

Closed nagaradhesh closed 2 years ago

nagaradhesh commented 2 years ago

We did two different memcached experiments in a CXL enabled host booted using SMDK kernel. For both, we ran memcached server on CXL exposed memory. First experiment, in default Zone partition setting, memcached server is run in exmem zone in compatible mode. Second experiment using numactl to redirect server traffic to CXL memory node1. We tried to plot Latency vs TPS(Transactions Per Second), we observed better latency numbers with Zone partition setting compared to numactl test for Node-1 CXL memory.

Can you please let us know if this is expected ? and is there any optimization in memory allocation for Zone partition. Following is the Configuration used: export CXLMALLOC_CONF=use_exmem:true,exmem_zone_size:12288,normal_zone_size:12288,maxmemory_policy:oom,use_auto_arena_scaling:false,priority:exmem

KyungsanKim commented 2 years ago

Hi, thank you for sharing us the experiment and result.

On your test condition, there are 2 SMDK contributions possibly related to the memcached performance.

  1. heap allocator : scalable DDR/CXL aware heap allocation
  2. free page traversal : inter-zone traversal is cheaper than inter-node traversal. But it looks minor on your bed.
nagaradhesh commented 2 years ago

Thanks for the response. "free page traversal : inter-zone traversal is cheaper than inter-node traversal." Can you please share more details on how page allocation is different b/w inter-zone and inter-node ? Please share if there are any links related to more information on the same.

KyungsanKim commented 2 years ago

Hello, nagaradhesh. Thank you for your interest. Please refer to the answer on a separate channel and let us know your thought.