Open afuller-TT opened 2 weeks ago
why would this issue be this team's responsiblity?
For context, this need comes from my GGML backend where GGML's unittests tests for the ability to write back into tensors generally (as a part of supporting KV Cache). Currently the only way to do this is
Path 2 is extremely slow and ugly. It'll be much better if there's native support. It's something that's bugging me almost since day 1 so it's on my list of stuff when people asked what to improve.
For context, this need comes from my GGML backend where GGML's unittests tests for the ability to write back into tensors generally (as a part of supporting KV Cache). Currently the only way to do this is
- If it happens to be writing a row in a 3D tensor - use the KV cache update method as it's a fast path
- Else, slice and dice the original tensor, Concat 4 pieces from the original 1 from the updated data. To form a new tensor
Path 2 is extremely slow and ugly. It'll be much better if there's native support. It's something that's bugging me almost since day 1 so it's on my list of stuff when people asked what to improve.
By "this team" I'm referring to software infrastructure, not metal project as a whole. We should either re-assign to metal runtime or TTNN.
@TT-billteng I was just trying to record the feedback we received so it didn't get lost. And figured we could figure out the right team afterwards.
From the community: