supports Llama 2 inference analysis

cli99 / llm-analysis

Latency and Memory Analysis of Transformer Models for Training and Inference

Apache License 2.0

343 stars 40 forks source link

Closed cli99 closed 1 year ago

cli99 commented 1 year ago

This PR add the following changes to support Llama 2 inference analysis:

add group query attention (GQA) support
add a gpu cost (in dollar per GPU hour) argument to the inference analysis interface
add "tokens per sec" and "cost per 1 k tokens" analysis
allow specified hbm memory bandwidth (in GB/s)