Open oluwabajio opened 3 years ago
jinay1991
commented
3 years ago With respect to https://github.com/deezer/spleeter/issues/354#issuecomment-740448009, here we request re-training of the Spleeter Models and also establishing process so that it will be easy to retrain model. May be establishing on-request/manual/release CI Job would be worth here.
agur
commented
3 years ago The following discussion belonged here:
https://github.com/deezer/spleeter/issues/354#issuecomment-740400158 740400158 https://github.com/deezer/spleeter/issues/354#issuecomment-740429543 740429543 https://github.com/deezer/spleeter/issues/354#issuecomment-740440487 740440487 https://github.com/deezer/spleeter/issues/354#issuecomment-740448009 740448009 https://github.com/deezer/spleeter/issues/354#issuecomment-740449932 740449932
@oluwabajio - thanks for filing it. Could you please check our recent review and rename this ticket to: "Spleeter Models for Tensorflow 2 and tflite (mobile)" - or update as needed?
@jinay1991 @VVasanth - sounds good progress, thanks. Would like to try and collaborate on getting this out. And will actively help if/when time allows.
For project tracking / mgmt - would progress manually merge to https://github.com/deezer/spleeter/ project when done?
Re: https://github.com/deezer/spleeter/issues/354#issuecomment-740448009:
Hi @jinay1991, appreciate the prompt response! Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?) If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now. And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.
Yes I agree with you on all points. You can find dataset on musDB dataset Please check and add LICENSE file as well with Trained Model for consumers to know about.
👍
Lets carry out all further discussion can be carried out on #477 .
👍
Re: https://github.com/deezer/spleeter/issues/354#issuecomment-740449932:
HI @agur, @jinay1991 - I am currently involved in porting Spleeter code into TF2.0 with the latest features...Following are the summary of the activities completed till date:
--> Have used musdb dataset for training purpose...as per Spleeter... --> Have reproduced the data processing pipeline, as similar to Spleeter --> Have recreated the model generation process with identical network but using TF2.0 features of GradientTape... --> I am abe to create the model and export it to TFLite format as well...
👍
--> Have built the android app that processes the data and feeds it to the TFLite model...there are still some work to be done and will complete them prob by today... --> You can find the github repository link below...
👍 , would like to see all those
--> I have trained the model only with less data and for only 2000 runs on my CPU (which ran for ~5 days)...Spleeter has been trained for 200K runs on the complete dataset on GPUs over a period of a week...so, accuracy of my model is pretty less...I will focus on training my model on Google Collab and improving the accuracy in coming week...
So, right now - I have the Spleeter version ported to TF2.0 with all its features and the Android app that leverages the generated TFLite model available...But the accuracy is less :-(
If you are interested, you can join with me and work on improving. the accuracy..thereby, we will be able to build an audio separation model deployed on edge...
👍 , we can try figure using the gpu on local laptop. 2013 MBP went from 2k to 22k on Geekbench when adding eGPU on both OpenCL and Metal (saw online) 2019 MBP i7 32gb AMD Radeon Pro 5300M Geekbench GPU gets 26.5k on both OpenCL and Metal. So I want to run the training and see.
Spleeter TF2.0 Model (Unofficial): https://github.com/VVasanth/SpleeterTF2.0_Unofficial Spleeter TF2.0 Mobile App Github link (Unofficial) : https://github.com/VVasanth/Spleeter_Unofficial_TF20_MobileApp
Pls let me know if you need any additional info and like to participate on this...
Let's keep collaborating. Would you please mind focus me on what to take from where to try training it? Since seeing few projects and getting a bit confused:
RufusMall
commented
3 years ago I have almost 0 ML Knowledge - iOS is my domain but I managed to port a version to iOS using: https://github.com/tinoucas/spleeter-tflite-convert
and jinay1991's repository as a reference (I believe he also helped answer some of my questions). One difference was that I used the C TFLite api instead of cpp.
I've got to say TFLite for iOS support feels a bit "beta" for this use (though maybe it is me just doing the wrong thing).
The binary size inflated to 800mb or something (because it includes all the OPS via TensorFlowLiteSelectTfOps. i think I have to recompile the SelectOPS library manually to solve this on iOS), and the memory usage/performance was not great. I got around the memory usage issues by just chunking the data from the audiofile before passing it to TFLite.
All this to say, if you guys want to build a demo iOS app I think I am happy to help.
One thing to note is that tflite also could not accelerate inference via GPU or neural engine, I believe someone else mentioned this was due to the dynamic tensor sizes. It might be worth keeping in mind this for future versions.
I can see there are apps out there using spleeter via PURE CoreML (bypassing tflite) which seems to be the superior solution. I have some ideas on how they might have done that - but not entirely sure since as i said I have 0 ML experience.
VVasanth
commented
3 years ago The following discussion belonged here:
#354 (comment) 740400158 #354 (comment) 740429543 #354 (comment) 740440487 #354 (comment) 740448009 #354 (comment) 740449932
@oluwabajio - thanks for filing it. Could you please check our recent review and rename this ticket to: "Spleeter Models for Tensorflow 2 and tflite (mobile)" - or update as needed?
@jinay1991 @VVasanth - sounds good progress, thanks. Would like to try and collaborate on getting this out. And will actively help if/when time allows.
For project tracking / mgmt - would progress manually merge to https://github.com/deezer/spleeter/ project when done?
Re: #354 (comment):
Hi @jinay1991, appreciate the prompt response! Should this discussion be on #477 then? (And should #477 title be "Spleeter Models for Tensorflow 2 and tflite (mobile)" rather than the existing: ~Spleeter Models for Tensorflow tflite~?) If the other routes are hacky and we're clear that re-train models for TF2 which is easy to convert to tflite is the right way than maybe we want to only focus on that now. And for 1.: Do we have the original data/music the original was trained with? So re-train can happen? And even if it is heavy/complex/won't-happen-too-often - we may want to ensure the process is as easy to repeat as possible for the future while running the current pass of re-train.
Yes I agree with you on all points. You can find dataset on musDB dataset Please check and add LICENSE file as well with Trained Model for consumers to know about.
👍
Lets carry out all further discussion can be carried out on #477 .
👍
Re: #354 (comment):
HI @agur, @jinay1991 - I am currently involved in porting Spleeter code into TF2.0 with the latest features...Following are the summary of the activities completed till date: --> Have used musdb dataset for training purpose...as per Spleeter... --> Have reproduced the data processing pipeline, as similar to Spleeter --> Have recreated the model generation process with identical network but using TF2.0 features of GradientTape... --> I am abe to create the model and export it to TFLite format as well...
👍
--> Have built the android app that processes the data and feeds it to the TFLite model...there are still some work to be done and will complete them prob by today... --> You can find the github repository link below...
👍 , would like to see all those
--> I have trained the model only with less data and for only 2000 runs on my CPU (which ran for ~5 days)...Spleeter has been trained for 200K runs on the complete dataset on GPUs over a period of a week...so, accuracy of my model is pretty less...I will focus on training my model on Google Collab and improving the accuracy in coming week... So, right now - I have the Spleeter version ported to TF2.0 with all its features and the Android app that leverages the generated TFLite model available...But the accuracy is less :-( If you are interested, you can join with me and work on improving. the accuracy..thereby, we will be able to build an audio separation model deployed on edge...
👍 , we can try figure using the gpu on local laptop. 2013 MBP went from 2k to 22k on Geekbench when adding eGPU on both OpenCL and Metal (saw online) 2019 MBP i7 32gb AMD Radeon Pro 5300M Geekbench GPU gets 26.5k on both OpenCL and Metal. So I want to run the training and see.
Spleeter TF2.0 Model (Unofficial): https://github.com/VVasanth/SpleeterTF2.0_Unofficial Spleeter TF2.0 Mobile App Github link (Unofficial) : https://github.com/VVasanth/Spleeter_Unofficial_TF20_MobileApp Pls let me know if you need any additional info and like to participate on this...
Let's keep collaborating. Would you please mind focus me on what to take from where to try training it? Since seeing few projects and getting a bit confused:
https://github.com/deezer/spleeter
- seems the main original project.
- VVasanth/SpleeterTF2 (2 projects)
jinay1991/spleeter
- Both of those are not forked, just standalone repos.
- Would we just convert them to a PR to deezer/spleeter eventually? (And if yes, should we fork one and do it there? Or if it does not matter now - sure - can progress as is on a standalone and figure PRing/merging back later on)
Thanks @agur...I am cleaning up the repo and adding the instructions to set the dev environment...you should be having the detailed instructions available by today...will keep this thread posted on this..Please let me know should we create any hitter or other chat platform for better collaboration...
agur
commented
3 years ago Thanks @RufusMall
I have almost 0 ML Knowledge - iOS is my domain but I managed to port a version to iOS using: https://github.com/tinoucas/spleeter-tflite-convert
and jinay1991's repository as a reference (I believe he also helped answer some of my questions). One difference was that I used the C TFLite api instead of cpp.
I've got to say TFLite for iOS support feels a bit "beta" for this use (though maybe it is me just doing the wrong thing).
Good that things work on iOS, and that it can be checked in parallel to Android. Believe once we'd figure the re-train the convert older model part won't be needed. Interesting if TFLite is mature on iOS or not yet. Looks like we're all learning.
The binary size inflated to 800mb or something (because it includes all the OPS via TensorFlowLiteSelectTfOps. i think I have to recompile the SelectOPS library manually to solve this on iOS), and the memory usage/performance was not great. I got around the memory usage issues by just chunking the data from the audiofile before passing it to TFLite.
Appears from this and related posts and approaches I heard that chunking to very small parts is needed due to resource constraints. And that gpu acceleration is a challenge to get to work. Would be handy to figure both direction, OpenCL and Metal were not only highly performant on my check (on laptop) as I wrote, it was also ~5 times better than without it thus promising. But given phone variations, even if that works on some it's going to be hard to depend on. And chunking, is annoying to manage but important to deal w/ properly as a base.
All this to say, if you guys want to build a demo iOS app I think I am happy to help.
👍
One thing to note is that tflite also could not accelerate inference via GPU or neural engine, I believe someone else mentioned this was due to the dynamic tensor sizes. It might be worth keeping in mind this for future versions.
Good one to check.
I can see there are apps out there using spleeter via PURE CoreML (bypassing tflite) which seems to be the superior solution. I have some ideas on how they might have done that - but not entirely sure since as i said I have 0 ML experience.
Interesting, so maybe pure CoreML is best for iOS, and tflite for Android. Wonder if it uses gpu well w/ CoreML, and also generally for all mobiles how strong / stable / mature is gpu on phones.
agur
commented
3 years ago Thanks @agur...I am cleaning up the repo and adding the instructions to set the dev environment...you should be having the detailed instructions available by today...will keep this thread posted on this..Please let me know should we create any hitter or other chat platform for better collaboration...
Thanks @VVasanth - look forward to hear when you are ready and try it soon.
Is your method the main effort to make deezer/spleeter TF2 compatible? Was it not already done elsewhere on the deezer/spleeter repo?
As to the multiple external repos which are related and helpful - maybe one of us can summarize and have it updated on the feature description.
agur
commented
3 years ago VVasanth
commented
3 years ago Thanks @agur...I am cleaning up the repo and adding the instructions to set the dev environment...you should be having the detailed instructions available by today...will keep this thread posted on this..Please let me know should we create any hitter or other chat platform for better collaboration...
Thanks @VVasanth - look forward to hear when you are ready and try it soon.
Is your method the main effort to make deezer/spleeter TF2 compatible? Was it not already done elsewhere on the deezer/spleeter repo?
As to the multiple external repos which are related and helpful - maybe one of us can summarize and have it updated on the feature description.
Hi @agur - Pls refer the below link, have provided the detailed information on the env setup and the progress made till now. Pls look at it and let me know if you need additional info.
I have repurposed the data processing and network architecture from deezer/spleeter repo. With regard to custom training logic, I have used the TF2.0's latest features and exported the model in 'savedmodel' format - so its straight forward to port it in to TFLite.
agur
commented
3 years ago Thanks @agur...I am cleaning up the repo and adding the instructions to set the dev environment...you should be having the detailed instructions available by today...will keep this thread posted on this..Please let me know should we create any hitter or other chat platform for better collaboration...
Thanks @VVasanth - look forward to hear when you are ready and try it soon. Is your method the main effort to make deezer/spleeter TF2 compatible? Was it not already done elsewhere on the deezer/spleeter repo? As to the multiple external repos which are related and helpful - maybe one of us can summarize and have it updated on the feature description.
Hi @agur - Pls refer the below link, have provided the detailed information on the env setup and the progress made till now. Pls look at it and let me know if you need additional info.
I have repurposed the data processing and network architecture from deezer/spleeter repo. With regard to custom training logic, I have used the TF2.0's latest features and exported the model in 'savedmodel' format - so its straight forward to port it in to TFLite.
@VVasanth - thanks, I believe you refer to the first of these links you sent before:
Spleeter TF2.0 Model (Unofficial): https://github.com/VVasanth/SpleeterTF2.0_Unofficial Spleeter TF2.0 Mobile App Github link (Unofficial) : https://github.com/VVasanth/Spleeter_Unofficial_TF20_MobileApp
I forked/cloned the first and will further look at it. I don't see a requirements.txt file - plz update if there is another way to get the packages consistently on a new virtual env.
As for the 4.4GB data - still awaiting the approval from the request yesterday. Guess it should be in soon.
VVasanth
commented
3 years ago @agur - Have added requirements.txt file with the required packages. Pls try and let me know. I feel, it would be better to have the further communication on Spleeter_Unofficial github - so that we can sort out any technical issues. Request you to continue the discussion over there. Thanks...
agur
commented
3 years ago @agur - Have added requirements.txt file with the required packages. Pls try and let me know. I feel, it would be better to have the further communication on Spleeter_Unofficial github - so that we can sort out any technical issues. Request you to continue the discussion over there. Thanks...
Thanks @VVasanth - can cont. there sure. What I am not clear about is how when this gets back into the deezer/spleeter. But can do that later maybe too.
agur
commented
3 years ago Also, quick refine my earlier performance points on the the laptop
👍 , we can try figure using the gpu on local laptop. 2013 MBP went from 2k to 22k on Geekbench when adding eGPU on both OpenCL and Metal (saw online) 2019 MBP i7 32gb AMD Radeon Pro 5300M Geekbench GPU gets 26.5k on both OpenCL and Metal. So I want to run the training and see.
It is actually 6 times faster and correlates to the 6 cores of the cpu. So it is likely a CPU parallelization and not a GPU one. The gpu I got is radeon-pro-5300m with 20 Compute Units.
Just that geekbench scores confused me. They show 6x improvement when I change from using Intel(R) UHD Graphics 630 to using AMD Radeon Pro 5300M Compute Engine. It seems the better graphic pick forces parallelization, but that happens in reality in cpu land and not gpu land.
Unfortunate it's not accessible to use the GPU, waste of good processing units.
And it could be too much to train on a laptop either way.
@VVasanth's and I keep collaborating on re-train on https://github.com/VVasanth/SpleeterTF2.0_Unofficial, encourage anyone who can help to chime in. And also may need access to stronger compute power soon when we get ready to run the re-train process.
agur
commented
3 years ago Hi Community,
Would like to solidly define what's needed for model management and have it reviewed by owners/community so we have a good buy-in and alignment of everyone since we'd be tackling it optimally.
There are 2 paths to progress here in parallel. Then have it clearly documented on the main project:
Copy owners - @Faylixe, @mmoussallam, @alreadytaikeune, @romi1502
Once we've revised and concluded on the best approach - we can update this ticket by @oluwabajio, and / or create tixs / update repos, as needed.
Thanks much!
agur
commented
3 years ago For the short term - does anyone have a usable decent quality model already converted to TF2? @jinay1991, @guomingjin, @tinoucas, or anyone gotten to something reusable? Or can anyone point us where to look besides efforts already mentioned here?
I will dig a little into jinay1991's convert repo based on tinoucas's start. Still hoping I don't have to do a convert and we can just inherit and share a decent existing one.
Copy: @VVasanth
agur
commented
3 years ago Happy Holidays everyone.
Repo owners - @Faylixe, @mmoussallam, @alreadytaikeune, @romi1502 - how do you suggest to review and ensure we are both effective and the community buys-in to the steps?
Anyone got a converted model with decent quality which can be shared as is?
@VVasanth and a few others keep progressing on Vasanth's current re-train suggested project.
theblissprogrammer
commented
2 years ago I've found a way to use pretrained models in Android / iOS APPs.
Mind sharing what you have?
luckysmg
commented
2 years ago Any updates?
hello, please can we get Spleeter Models for Tensorflow tflite. Or if the functionality is not yet available, how can we convert the current model to be tflite compartible