search

microsoft / vscode-ai-toolkit

MIT License
1.23k stars 65 forks source link

Support for Copilot+ PCs #92

Open pkbullock opened 2 months ago

pkbullock commented 2 months ago

It would be great to see if AI Toolkit can leverage the NPU in Copilot PCs. Currently this uses the CPU, its nice a quick on the Snapdragon processors but not using the AI processor when running models.

sirredbeard commented 2 months ago

I wonder if this is related to onnxruntime-genai still awaiting QNN support.

pkbullock commented 2 months ago

This is listed in the docs as supports AI Copilot PC but it doesnt, my NPU activity is 0%. So how to use this?

sirredbeard commented 2 months ago

I don't see any reference yet to CoPilot+ PC in the AI Toolkit docs, at least not here. Because it relies on onnxruntime-genai, I believe QNN support must land there first before AI Toolkit can take full advantage of it. You might be able to take some advantage of the NPU now, indirectly, by using DirectML with a model like Phi-3-mini-4k-directml-int4-awq-block-128-onnx which is optimized for that. I have been using DirectML on my non-CoPilot Qualcomm-based WDK23 to speed up training.

pkbullock commented 2 months ago

Hi @sirredbeard - I saw it in the release notes on installation of the VSCode extension with the mention of support. But I agree seems many frameworks are dependent on the QNN runtimes/sdks being release. image

wmmc88 commented 2 months ago

It seems like direct-ml models don't show up in the model catalog on my PC that has a QC NPU

rockcat commented 4 weeks ago

Me neither - what is the course of action to enable models to show up on Snapdragon machines ?