intel / xFasterTransformer

Apache License 2.0
270 stars 53 forks source link

Add env param KV_CACHE_LOCATION to control kv cache memory numanode location #462

Open a3213105 opened 1 week ago

a3213105 commented 1 week ago

Usage: before you run instance export KV_CACHE_LOCATION=#memory_numa_node_id_you_want_to_use_for_kv_cache

by defaults, kv_cache location is the same as other parts of instance.

pujiang2018 commented 1 week ago

@a3213105 we need to put the environment reading to a unified place, please put it to environment.h

a3213105 commented 1 week ago

@a3213105 we need to put the environment reading to a unified place, please put it to environment.h

DONE