search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Set physical cpu affinity for micro case #656

Open Qinghe12 opened 3 years ago

Qinghe12 commented 3 years ago

env info: hadoop 2.6.5 + spark 2.4.6 + Hibench 7.1 physical machine with 128 physical cores： NUMA node0 CPU(s): 0-31 NUMA node1 CPU(s): 32-63 NUMA node2 CPU(s): 64-95 NUMA node3 CPU(s): 96-127

Can I bind task to physical core 0-3 when running wordcount case with following spark.conf？

hibench.yarn.executor.num 1 hibench.yarn.executor.cores 4

minmingzhu commented 3 years ago

@Qinghe12 Can you explain this more details ? What I understand is you have four nodes and each node has 32 physical CPUs, you want to use physical core 0-3 for each nodes when running wordcount case? A total of 16 physical cores are used for this task？ hibench.yarn.executor.num This parameter indicates the total number of executors for your task , hibench.yarn.executor.cores this parameter indicates the number of cores used by each executor, So the total number of cores you use should be hibench.yarn.executor.num * hibench.yarn.executor.cores.

Qinghe12 commented 3 years ago

yes，I only want to use physical core 0-3 in node 0 when running wordcount case with hibench.yarn.executor.num=1 and hibench.yarn.executor.cores=4. Is there any way to bind to core 0-3 when running wordcount?

minmingzhu commented 3 years ago

@Qinghe12 Setting hibench.yarn.executor.num=1 and hibench.yarn.executor.cores=4 that can use four physical cores, but which core to use is random.