How to visualize/check IPU utilization while executing Llama2 on AMD?

amd / RyzenAI-SW

MIT License

403 stars 65 forks source link

How to visualize/check IPU utilization while executing Llama2 on AMD? #99

Closed AshimaBisla closed 4 months ago

AshimaBisla commented 4 months ago

Hello, I am trying to rum Llama2 model on AMD using the given github:https://github.com/amd/RyzenAI-SW/blob/main/example/transformers/models/llama2/README.MD

When I choose either "AIE" or "cpu" as target and check my task manager while execution, I see CPU utilization only(going upto 50-70%). Is there any way to understand and visualize APU/IPU utilization when I choose "aie" as my target?

Thanks, Ashima

uday610 commented 4 months ago

You should see a tokens/sec difference when you target CPU vs NPU...

We will release a new driver, which will help visualize the NPU load through Windows Task Manager.