Closed philipturner closed 1 year ago
Yeah, there are more issues with MPSMatrixSoftMax than I expected, probably need to rewrite that to MSL. lib/nnc/cmd
should get the most changes nowadays, as the core (autograd, graph scheduling, tensor allocation) are fairly stable now (the only change I planned is to support "freeze" the model so that during backprop, the gradients of the frozen model will be discarded to reduce memory usage (quite important to implement "textual inversion" or "hypernetwork")).
Very happy to see this ported to Metal. Now that I've given up on S4TF, it would be interesting to learn of a Swift framework (s4nnc) that runs deep learning on Metal. Regardless, I'll probably use MPSGraph directly in my future work instead of a high-level API.
I haven't seen the
nnc
directory of this repo get changes, but will that be ported to Metal soon? I also haven't seen Metal shader files yet, while there are custom CUDA shaders. Don't be afraid to use MSL for operations not supported by MPSGraph.