search

jatinchowdhury18 / RTNeural

Real-time neural network inferencing
BSD 3-Clause "New" or "Revised" License
605 stars 61 forks source link

Support for TCN #120

Open robjlyons opened 12 months ago

robjlyons commented 12 months ago

Hi, I have been using TCNs to train and model more complicated effects like Delay and Reverb (https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Farxiv.org%2Fabs%2F2112.02926)

I have also had some luck in using these in the Neutone plugin, however, it's not realtime. There is a long delay making it unplayable.

Wondering whether it would be possible to support TCN trained models or neutone models directly?

Thanks,

jatinchowdhury18 commented 12 months ago

Hello!

Yes, it is definitely possible to implement a TCN model (as described in the paper you linked) within RTNeural. I had done an implementation a few months ago, but haven't pushed it anywhere yet since it hasn't really been tested. For that i remember having to implement a FiLM layer for myself, as well as a TCNBlock which has some non-sequential routing in it. I'm happy to dig up that code and discuss it more if you like!

If you want to have a look at https://github.com/jatinchowdhury18/RTNeural/pull/116, @purefunctor has also implemented a "MicroTCN" network. We should have that PR merged in the next couple of days.

robjlyons commented 12 months ago

I would be very interested in seeing any code you have for TCN networks, the FiLM layer is something I hadn't even gotten to yet so any help would be amazing. Thank you. In the meantime I'll take a look at 116.

jatinchowdhury18 commented 12 months ago

Sure thing! I just uploaded my code here: https://github.com/jatinchowdhury18/RTNeural-Experimental/blob/main/examples/snafx/SNAFxModel.h

As I had mentioned, it's not very well tested, but hopefully the overall ideas there are useful!