jatinchowdhury18
commented
2 months ago Hello, thanks for making the issue. These are good ideas, but they are also part of a larger discussion about how RTNeural can/should inter-operate with the libraries that people are using for training neural networks, so I'm going to provide a bit of background. I hope I'm not re-explaining things that you already know, but I just want to be sure that we're on the same page.
When I started working on RTNeural, I was starting with a "sequential" model trained in Tensorflow. That's where the Python script for "exporting" a TensorFlow model as a JSON file originated. While this script still works and is useful, it has several limitations, most notably, it only works with models trained in TensorFlow, and it only works with neural networks that have a "sequential" architecture.
One area of confusion is that there are other ways (aside from the one provided by RTNeural) to convert a trained neural network to a JSON file. For example, the GuitarML project often uses networks that were trained in PyTorch, and use a different approach for exporting the model weights as a JSON file (RTNeural uses a similar approach for testing inter-operability with PyTorch). However, the JSON files exported from PyTorch are not directly compatible with the ones used by RTNeural.
So when loading a trained network into RTNeural, the README demonstrates the "simplest" method, but also most limited method. As you noticed, all of the relevant "layer" implementations in RTNeural contain methods for setting the weights used by that layer. While these methods are "optional" for folks who are loading their models from an RTNeural-compatible JSON file, they are kind of "mandatory" for everyone else. So the most "general" way to load weights into an RTNeural model is to do something like this (which I believe is similar to your example code):
RTNeural::ModelT<float, 2, 1, RTNeural::LSTMLayerT<float, 2, hidden_size>, RTNeural::DenseT<float, hidden_size, 1>> model;
auto& lstm = model.get<0>();
lstm.setWVals(...);
lstm.setUVals(...);
lstm.setBVals(...);
auto& dense = model.get<1>();
dense.setWeights(...);
dense.setBias(...);I think that's about everything I wanted to mention about the past and present states of RTNeural. Now thinking about the future: what should we do differently, and how can we do better overall?
I think the most obvious thing that needs to improve is the documentation. The weights-loading methods are often poorly-named, and are not always documented in such a way that the user can easily tell, for example, what should be the dimensions of the data being passed into this method?
I think the next obvious thing is making the weights-loading methods more flexible. Right now most of these methods accept either a std::vector<T> or std::vector<std::vector<T>>. Having to use a vector can be inefficient, not to mention that I think std::vector<std::vector<T>> is a very poor way to represent a 2D "matrix" (unfortunately I didn't know any better at the time). Maybe we can take some inspiration from newer C++ features like std::span and std::mdspan here.
Anyway, getting back to the most immediately relevant problem: how can RTNeural support users in getting their trained network weights from their training environment into RTNeural? There's two parts to this: "exporting" the weights from the training environment, and "importing" the weights into RTNeural.
For "exporting" weights, I'm having trouble thinking of any RTNeural-driven solution that can be robust in meeting all users' needs. Even with the relatively simple script that RTNeural currently supports for exporting TensorFlow sequential models, the script sometimes needs updating when TensorFlow changes some of its internals, and there's still a number of "edge cases" that the script doesn't handle. If we were to try making that a "general" solution, by adding support for non-sequential models, PyTorch models, and multiple "output formats" (e.g. JSON, C-style header file, etc), I worry that the effort of maintaining that solution would end up being larger than the effort required to maintain the rest of RTNeural, and would distract from the "actual" goals of this project. That said, if a solution like this were to exist (or already exists) outside of RTNeural, that could be very useful for us!
So with that in mind, I think the best approach would be to work with the "exporting formats" provided by the major training libraries. We've been doing a little bit of work in this direction already, with having some helper functions to load RTNeural layers using the JSON representation exported by PyTorch. That said, I don't think that TensorFlow or PyTorch automatically support exporting header files, and I agree that would be a special case that RTNeural should try to provide support for. Maybe there could be some RTNeural "extension" to generate a header file using the work that RTNeural is already doing to load weights for some layers? Otherwise, it would mostly be up to the user to figure out how to convert the weights exported from their training environment into a header file, if that's something that they want to do (as the GuitarML folks did in the example you linked to).
For "importing" network weights on the RTNeural side, I've been gradually less in favor of an "automatic" solution, which is sort of what we have now with the parseJson() methods. While these methods can be convenient for simple network architectures, I really don't think they're a good solution for more general cases. But the alternative is that people have to mostly figure out for themselves how to get their weights into their RTNeural model, which would be less convenient.
There's kind of a theme here, which I think touches a few other parts of RTNeural as well: to what extent do we want to provide "out-of-the-box" solutions, and to what extent do we want to provide a more "open" and flexible interface that users may have to navigate more on their own? I think the best solution is to try doing both, so that the library is easy to use for beginners, but also very capable for "power users"... unfortunately trying to do both takes effort :).
I think that's everything I wanted to say... sorry to ramble on for so long! Thinking more on it, I like the idea of an RTNeural layer being able to return a string containing it's own weights, which could then be used to construct a header file. I'll think more about this, and maybe try to make a small example of how this could work!
jake-is-ESD-protected
RTNeural seems like a great choice for resource constrained devices, which really makes me wonder why loading a model requires a file system.
According to the docs, the compile time API can only load a model's weights via a
jsonfile, which requires a file system. While file systems can be implemented on embedded systems such as the Daisy Seed, a perfect target device for RTNeural, it seems like an unnecessary stretch to add one just for loading a simple model. Instead, ajsonto.hparser would create a.hfile at compile time which would directly be linked to the firmware, eliminating any form of file system dependency. Or the.jsonfile is bypassed directly and a saved model can be converted to a header file via a script.GuitarML did this with a manual python-based parser, but it is very unclear to me how the developers got the information on the necessary syntax, as the example is tailored exactly for their application and nothing more. I believe that a modular solution directly from you would greatly benefit applications for embedded systems and could easily outperform tflite micro or X-CUBE-AI, among others.
It would already be helpful to have some docs on the required "even more static" initialization of models by directly assigning weights as arrays to layers of the model. If they exist and I missed them, I'd be happy to be pointed to them by you. I noticed that I can
setWeights()of a single layer at runtime, but that is missing a way to add that layer to the model. Would be nice if something like this would be possible:This way, the architecture is still set up statically, while the weights are loaded in as
constarrays from somewhere else. The indexing management of the weights would drop back to the user. While this might lead to unintended misuse, a helper script as mentioned above might be a good addition to parse a.jsonor saved model file into RTNeural-standardized constant shapes.