search

haoheliu / AudioLDM

AudioLDM: Generate speech, sound effects, music and beyond, with text.
https://audioldm.github.io/
Other
2.45k stars 222 forks source link

audio-to-audio: is it possible to use more than one samples? #112

Open buscon opened 1 year ago

buscon commented 1 year ago

for sound design purposes, it would be interesting to use the audio-to-audio feature with multiple samples

is that possible? if not, any hint on how I could add such feature?

ivcylc commented 1 month ago

can you describe your request in detail? maybe i can help implement it

buscon commented 1 month ago

thanks!

This is what I imagine:

  1. the user provide as a prompt different samples, the output will be a combination of these samples. it is not a simple mix of different sounds, but more each sound merges and influence the other.
  2. the prompt should include how much influence each prompt sample have, something like audioldm --mode "transfer" --file_path trumpet.wav 70% cello.wav 30% -t "Children Singing" if no percentage is given, each prompt sample should have the same influence on the output. I think you are doing something similar, mixing the influence of the audio prompt together with the influence of the text prompt
  3. as an extra feature, it'd be useful to have some extra audio parameters, similar to what you have in synthetizers, like filters, eqs, effects to influence the output. But this is something for later on.
ivcylc commented 1 month ago

i have a lot of ideas to implement your idea , wait my arXiv paper this year (i am doing something other currently)

Tortoise17 commented 1 month ago

@buscon does transfer learning from the input audio sample for human spoken to generate a specific human voice with model with giga_speech already works?

buscon commented 1 month ago

@buscon does transfer learning from the input audio sample for human spoken to generate a specific human voice with model with giga_speech already works?

I think it does, though I tested it long time ago and cannot remember right now. I will try it again soon and report back here.

Tortoise17 commented 1 month ago

@buscon did you manage to find the way? or any success?

buscon commented 1 month ago

@buscon did you manage to find the way? or any success?

not yet, I cannot install audioldm with pip anymore. I think it's related to the overall upgrade of python 3.12. I will answer again when I figured that out.