search

jatinchowdhury18 / RTNeural

Real-time neural network inferencing
BSD 3-Clause "New" or "Revised" License
551 stars 57 forks source link

Add support for NAM files #143

Open christoph-hart opened 2 weeks ago

christoph-hart commented 2 weeks ago

Hi Jatin,

how hard would it be to add support for parsing the NAM file format?

https://github.com/sdatkinson/NeuralAmpModelerCore

Just from a quick peek at both sources the required layers are almost there (except for the wavenet layer which seems like a high level abstraction of existing low level layers).

I would love to avoid adding too many neural network engines to my project so if you think it‘s doable I‘ll give it a shot.

brummer10 commented 2 weeks ago

@christoph-hart I've created a project were I implemented both engines. So users could load nam and json/aidax files without take care which one to load. Implementation of both engines is straight forward. https://github.com/brummer10/Ratatouille.lv2

christoph-hart commented 2 weeks ago

Thanks for the input and adding both engines is definitely an option but I would love to avoid adding the big fat Eigen library and RTNeural is already in there with what looks to me 95% of the required feature set.

jatinchowdhury18 commented 2 weeks ago

Hi All!

I think it should be possible to construct a NAM-style model using RTNeural's layers. If I remember correctly NAM uses a "Temporal Convolutional Network", and I have implemented a couple of those in the past using RTNeural's layers, although there are sometimes variations between those types of networks. Here's an example of a "micro-TCN" implementation that we use as part of RTNeural's test suite. Probably the best route forward would be to use that implementation as a starting point, add whatever might be missing from the NAM model architecture, and probably adapt the mechanism for loading model weights to match whatever format NAM models use to store their weight. I'd be happy to help with this process as my time allows.

That said, I'm not sure it would make sense to add support for NAM models directly to RTNeural, since I think it falls a little bit outside the scope of what RTNeural does. I do have some future plans for a sort of "model library" which would have example implementations of several neural network architectures that are commonly used in real-time audio (and maybe other real-time domains as well), and I think having NAM models as part of the model library would be great. However, there's some other changes I want to make to RTNeural before starting on that, so it may be a while before I get there.

olilarkin commented 2 weeks ago

Also interested in this. https://github.com/sdatkinson/NeuralAmpModelerCore/issues/49

RustoMCSpit commented 1 week ago

maybe relevant? https://github.com/Chowdhury-DSP/BYOD/issues/363

brummer10 commented 1 week ago

Thanks for the input and adding both engines is definitely an option but I would love to avoid adding the big fat Eigen library and RTNeural is already in there with what looks to me 95% of the required feature set.

Just out of curiosity I checked if we could build NeuralAmpModelerCore against the Eigen library comes with RTNeural, and yes, it works flawless. We could even share the jsaon header.