search

microsoft / Windows-Machine-Learning

Samples and Tools for Windows ML.
https://docs.microsoft.com/en-us/windows/ai/
MIT License
1.03k stars 446 forks source link

Dynamic axis size zero not supported #512

Closed tom-huntington closed 1 year ago

tom-huntington commented 1 year ago

I'm doing key value caching for openai whisper, and am concatenating the new key values onto the old kv cache along a dynamic axis representing the token length.

On the initial pass, the dynamic axis would be size zero. Which is fine in pytorch, 

>>> torch.cat((torch.zeros(2, 0, 3), torch.zeros(2,1,3)), dim=1).shape
torch.Size([2, 1, 3])

but raises a exception

C:\apilot\agent_work\6\s\engine\lotus\winml\lib\api\impl/TensorBase.h(102)\Windows.AI.MachineLearning.dll!00007FFB751AFAB2: (caller: 00007FFB751B0038) Exception(1) tid(227c) 80070057 The parameter is incorrect.

when calling LearningModelBinding::Bind in WinML:

binding.Bind(L"kv_cache", TensorFloat16Bit::Create({ 24, 1, 0, 768 }));

Just letting you know...