search

jatinchowdhury18 / RTNeural

Real-time neural network inferencing
BSD 3-Clause "New" or "Revised" License
543 stars 57 forks source link

Support for TCN #120

Open robjlyons opened 7 months ago

robjlyons commented 7 months ago

Hi, I have been using TCNs to train and model more complicated effects like Delay and Reverb (https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Farxiv.org%2Fabs%2F2112.02926)

I have also had some luck in using these in the Neutone plugin, however, it's not realtime. There is a long delay making it unplayable.

Wondering whether it would be possible to support TCN trained models or neutone models directly?

Thanks,

jatinchowdhury18 commented 7 months ago

Hello!

Yes, it is definitely possible to implement a TCN model (as described in the paper you linked) within RTNeural. I had done an implementation a few months ago, but haven't pushed it anywhere yet since it hasn't really been tested. For that i remember having to implement a FiLM layer for myself, as well as a TCNBlock which has some non-sequential routing in it. I'm happy to dig up that code and discuss it more if you like!

If you want to have a look at https://github.com/jatinchowdhury18/RTNeural/pull/116, @purefunctor has also implemented a "MicroTCN" network. We should have that PR merged in the next couple of days.

robjlyons commented 7 months ago

I would be very interested in seeing any code you have for TCN networks, the FiLM layer is something I hadn't even gotten to yet so any help would be amazing. Thank you. In the meantime I'll take a look at 116.

jatinchowdhury18 commented 7 months ago

Sure thing! I just uploaded my code here: https://github.com/jatinchowdhury18/RTNeural-Experimental/blob/main/examples/snafx/SNAFxModel.h

As I had mentioned, it's not very well tested, but hopefully the overall ideas there are useful!