search

Neutone / neutone_sdk

Join the community on Discord for more discussions around Neutone! https://discord.gg/VHSMzb8Wqp
GNU Lesser General Public License v2.1
470 stars 22 forks source link

TensorFlow support #2

Closed vvolhejn closed 2 years ago

vvolhejn commented 2 years ago

Hi, exciting project! I can see that the SDK is built for PyTorch models. I'm developing a model on top of DDSP which is in TensorFlow. Is there any chance of me being able to get the model onto Neutone now or in the future? Thanks!

bogdanteleaga commented 2 years ago

Hello! Thank you for your interest. There are no plans of supporting TensorFlow for the time being. I'd recommend trying to use one of the DDSP PyTorch implementations if possible.

vvolhejn commented 2 years ago

For completeness, here's the response I got from @Fyfe93 about this. I formulated the question a bit more generally:

Hey, cool plugin. I wanted to ask whether you are planning support for non-Torch models. It would be great to be able to run e.g. ONNX or TVM models, which are framework-neutral. I would also be willing to help with the implementation 🙂

He responded:

Hey @vvolhejn ! Cheers for your kind words! So we may migrate to use ONNX or TVM in future. We considered this initially before starting the design but the issue we had was that a lot of the neutral frameworks can’t handle or translate the bleeding edge functionality and a bunch of the models we utilise require this. For example, at the time we decided, ONNX couldn’t handle STFTs etc. It would be great to be able to dynamically switch to different inference runtimes depending on the model type loaded but I fear the overall binary/bundle size would be ginormous. However, we should definitely keep this discussion open!

Would really love to see this!

Fyfe93 commented 2 years ago

Hey @vvolhejn ! I think on further reflection even though it would be great to support multiple frameworks there would need to be significant demand before we'd act on this as our current SDK format is based around PyTorch. I think for now I would echo @bogdanteleaga advice and recommend using a PyTorch model implementation. We may pickup cross-framework support in the future if it's feasible 😊