search

deezer / spleeter

Deezer source separation library including pretrained models.
https://research.deezer.com/projects/spleeter.html
MIT License
25.74k stars 2.82k forks source link

[Feature] TFLite (Pretrained Models) #354

Closed jinay1991 closed 3 years ago

jinay1991 commented 4 years ago

Description

As the repository provides, Tensorflow v1.15.2 Checkpoints which can be converted to SavedModel for all the pretrained models.

This issue is to request support for TFLite and support for TensorFlow v2.1.

Additional information

mmoussallam commented 4 years ago

Hi @jinay1991

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

waqasakram117 commented 4 years ago

Hi @mmoussallam Please make compatible with TF lite as well so it can target a vast variety.

Vnicius commented 4 years ago

This will be amazing

abhi-rawat1 commented 4 years ago

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

@mmoussallam Eagerly waiting of this to get out. Do we know when TFLite version will be available?

searchff commented 4 years ago

Description

As the repository provides, Tensorflow v1.15.2 Checkpoints which can be converted to SavedModel for all the pretrained models.

This issue is to request support for TFLite and support for TensorFlow v2.1.

Additional information

  • Current version of trained model (i.e. SavedModel) can not be converted to TFLite due to issues. This feature support is also indented to fix that issue and possibly retrain the model with TF v2.1

Would you please share the tflite file?

VVasanth commented 4 years ago

@mmoussallam - Would be really great if you could throw some light on when TF2.0 would be available.

guomingjin commented 4 years ago

Hi everyone, I have already converted the models into c++ implementation with android jni project, which doesn't depend on any deep learning framework. Those models are really great and pretty cool. I am arranging to make a real-time mobile app. Thank you for your great work. Regards.

waqasakram117 commented 4 years ago

@guomingjin It is great so, can you please share your amazing work with us.

actuy commented 4 years ago

Hi everyone, I have already converted the models into c++ implementation with android jni project, which doesn't depend on any deep learning framework. Those models are really great and pretty cool. I am arranging to make a real-time mobile app. Thank you for your great work. Regards.

@guomingjin That's amazing! Could you share this with us?

jinay1991 commented 4 years ago

Hi @jinay1991

Point taken, we'll investigate porting the model to TF2 compat in the coming weeks.

@mmoussallam Is there any update on this? Is anyone working on this?

mmoussallam commented 3 years ago

Closing this since v2 is now TF2 compatible

jinay1991 commented 3 years ago

@mmoussallam Thanks for taking care of support for TF2. Will you be kind to update pre-trained models as well with recent Spleeter Version?

agur commented 3 years ago

@mmoussallam Thanks for taking care of support for TF2. Will you be kind to update pre-trained models as well with recent Spleeter Version?

Is all needed for TFLite support is to re-train after the TF2 support was coded?

Is the progress here https://github.com/jinay1991/spleeter/issues/1 needed? Ready?

Can we please figure the needed steps together? Would be happy to help make it easy to use on Mobile.

jinay1991 commented 3 years ago

Hello @agur Thanks for your msg. We have a open feature request for the same https://github.com/deezer/spleeter/issues/477.

With the help of tinoucas/spleeter-tflite-convert, I was manage to convert all the existing models (TF v1.13) in a hacky way (i.e. Input/Output Tensors did not have their shapes defined in TFLite model hence ResizeTensors was necessary for TFLite Models afterwards and in addition to that binary for TFLite is comparatively too big due to the usage of TF_SELECT_OPS). Hence I am assuming that with retraining of the Model with TFv2 support would atleast avoid this hacky way and one could use Quantisation and Optimisation techniques to make binary smaller yet have same results.

  1. Retrain and Release Models with TFv2 versions
  2. Once TFv2 models are available in fact it shall be easy to release TFLite Models as well.
  3. If possible, remove TF OPS and use compatible OPS between TF/TFLite OPS. ops_compatibility
  4. Provide post-training quantisation for both types of models post_training_quantization, this requires training dataset hence it is best is someone who is training the model provides this thing as well.

And I think these are all we need for now in order to integrate this with any platform. Once done one can check/deploy with https://github.com/jinay1991/spleeter/tree/master/spleeter/test

agur commented 3 years ago

Hi @jinay1991, appreciate the prompt response!

Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?)

If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now.

And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.

jinay1991 commented 3 years ago

Hi @jinay1991, appreciate the prompt response!

Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?)

If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now.

And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.

Yes I agree with you on all points. You can find dataset on musDB dataset Please check and add LICENSE file as well with Trained Model for consumers to know about.

Lets carry out all further discussion can be carried out on #477 .

VVasanth commented 3 years ago

HI @agur, @jinay1991 - I am currently involved in porting Spleeter code into TF2.0 with the latest features...Following are the summary of the activities completed till date:

--> Have used musdb dataset for training purpose...as per Spleeter... --> Have reproduced the data processing pipeline, as similar to Spleeter --> Have recreated the model generation process with identical network but using TF2.0 features of GradientTape... --> I am abe to create the model and export it to TFLite format as well... --> Have built the android app that processes the data and feeds it to the TFLite model...there are still some work to be done and will complete them prob by today... --> You can find the github repository link below... --> I have trained the model only with less data and for only 2000 runs on my CPU (which ran for ~5 days)...Spleeter has been trained for 200K runs on the complete dataset on GPUs over a period of a week...so, accuracy of my model is pretty less...I will focus on training my model on Google Collab and improving the accuracy in coming week...

So, right now - I have the Spleeter version ported to TF2.0 with all its features and the Android app that leverages the generated TFLite model available...But the accuracy is less :-(

If you are interested, you can join with me and work on improving. the accuracy..thereby, we will be able to build an audio separation model deployed on edge...

Spleeter TF2.0 Model (Unofficial): https://github.com/VVasanth/SpleeterTF2.0_Unofficial Spleeter TF2.0 Mobile App Github link (Unofficial) : https://github.com/VVasanth/Spleeter_Unofficial_TF20_MobileApp

Pls let me know if you need any additional info and like to participate on this...

agur commented 3 years ago

Answered at: https://github.com/deezer/spleeter/issues/477#issuecomment-740527178, we can continue there.

ziad-halabi9 commented 3 years ago

@Turing311 can you please share your work?

KangChou commented 2 years ago

大家好，我已经用android jni项目将模型转换成c++实现，不依赖任何深度学习框架。 这些模型真的很棒而且很酷。 我正在安排制作一个实时移动应用程序。 感谢你杰出的工作。 问候。

Hello, as you said spleeter is implemented in C++, can you open source your project? Because I am currently doing research in this area.

KangChou commented 2 years ago

你好@agur感谢您的消息。我们对相同的#477有一个开放的功能请求。

在tinoucas/spleeter-tflite-convert的帮助下，我设法以一种骇人听闻的方式转换了所有现有模型（TF v1.13）（即输入/输出张量没有在 TFLite 模型中定义其形状，因此ResizeTensors对于之后的 TFLite 模型，此外，由于使用了 TF_SELECT_OPS，TFLite 的二进制文件相对较大）。因此，我假设通过对具有 TFv2 支持的模型进行再训练，至少可以避免这种骇人听闻的方式，并且可以使用量化和优化技术来使二进制文件更小，但结果相同。

  1. 使用 TFv2 版本重新训练和发布模型
  2. 事实上，一旦 TFv2 模型可用，也应该很容易发布 TFLite 模型。
  3. 如果可能，移除 TF OPS 并在 TF/TFLite OPS 之间使用兼容的 OPS。ops_compatibility
  4. 为两种类型的模型post_training_quantization提供训练后量化，这需要训练数据集，因此最好是训练模型的人也提供这个东西。

我认为这些就是我们现在所需要的，以便将其与任何平台集成。完成后，可以使用https://github.com/jinay1991/spleeter/tree/master/spleeter/test检查/部署

How do I compile cpp programs?