Open Leekinxun opened 1 month ago
@Leekinxun Hello there! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's work together to solve this issue.
To ensure that the create_blob_message
method returns the actual audio blob instead of just a link, you need to verify that the ToolInvokeMessage
is correctly configured to handle the blob data. The code you provided seems correct, but let's ensure everything is set up properly.
Here's a refined version of your code:
from core.tools.entities.tool_entities import ToolInvokeMessage
def call_tts_stream(text: str) -> Iterator[bytes]:
tts_url = url
tts_headers = build_tts_stream_headers()
tts_body = build_tts_stream_body(text)
response = requests.request("POST", tts_url, stream=True, headers=tts_headers, data=tts_body)
for chunk in response.raw:
if chunk:
if chunk[:5] == b'data:':
data = json.loads(chunk[5:])
if "data" in data and "extra_info" not in data:
if "audio" in data["data"]:
audio = data["data"]['audio']
yield audio
def audio_play(audio_stream: Iterator[bytes]) -> bytes:
audio = b""
for chunk in audio_stream:
if chunk is not None and chunk != '\n':
decoded_hex = bytes.fromhex(chunk)
mpv_process.stdin.write(decoded_hex) # type: ignore
mpv_process.stdin.flush()
audio += decoded_hex
return audio
audio_chunk_iterator = call_tts_stream(query)
audio = audio_play(audio_chunk_iterator)
return self.create_blob_message(blob=audio, meta={'mime_type': 'audio/mpeg'})
Ensure that the create_blob_message
method in the Tool
class is implemented as follows:
def create_blob_message(self, blob: bytes, meta: dict = None, save_as: str = '') -> ToolInvokeMessage:
"""
Create a blob message.
:param blob: The blob data.
:param meta: Metadata associated with the blob.
:param save_as: A string indicating how the blob should be saved.
:return: The blob message.
"""
return ToolInvokeMessage(
type=ToolInvokeMessage.MessageType.BLOB,
message=blob,
meta=meta,
save_as=save_as
)
This method should correctly return the blob message containing the audio data. If you are still seeing a link instead of the actual blob, ensure that the ToolInvokeMessage
class and its handling in your application are correctly set up to process and return blob data [1][2].
To continue talking to Dosu, mention @dosu.
@dosu Thanks for your response.And I have reviewed the two sections you mentioned and confirmed that the backend input content is as follows:
Outputs: [ToolInvokeMessage(type=<MessageType.BLOB: 'blob'>, message=b'ID3\x04\x00\x00\x00\x00\x00#TSSE\x00\x00\x00\x0f\x00\x00\x03Lavf58.29.100\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xfb\x98\xc4\x00\x00\xdcz\x1e\xfe\x07\xb0\xd7\xc3\xb0\xbd\xa4\xd0\xf3=y\x0f@\xdc\x88A\x12)x\x90\x0eB\x16\x85\xb7\xbb%\x08\x08\xf4\x91\x9f)\xf5^\xd8\xd9\xd4\xe7Z\xb1Y\xacJ\x87\xb7!\x0cj\xc8\xb3\xa4\xc8Y\x08\x02(\x1c\x020\x10j\xae\x11-\xf4\x84\xc1,\x1b\x93\xc4\xb3\xf6\xcf\xca\x88\x8c\x1d\x8db\xc5\x8c\x9f\xb0%\xab\xe0H\x8b!~\xcb\x16\x1c\x1c\x183fD\x83\x01!(7\x00q|G\xed\x87c0h\x8a\x10Lw\xcf\xbd\xd2\x13\x1cC\x12\xdf*\x13\x19yd+\xd7\xbfs\x84\xa2[Q>\x90\x98\x7f\x1d7\xe9\xb6q\x85\x8e^[~\xf7
\x90L\x8a\xff\x8b\x0f\x1f\x9b\xdf\xd800\xa9\x85`<l\xfdy\x9a\xe3x\x89fb[\xe7vd\x1a(\xca\x1c\x18\x18E\x82\x19o7\xdb_g>\xf0\x18E\x01\xc2\xc72\x9dv\xc9g\xf9\x87\x93\xbed\x104-\xf4\xb2u\x18\xfa\xd1\x1e\xf7/t\xc8\xbd\x8c\xfd\xa1\xfb\xdf1\r\xa8\x16\xc8\x10\x00\x17\x88\x19\x14\xed\x8a\xcaC\x9d\xfch\x04\x17\x82\x0b\x01\x90 \x87{)\x08\x
However, the return information is still a link : Link: /files/tools/b4dd3442-0f99-4072-9a65-40bd32e8bbdb.mp3
To ensure that the create_blob_message
method returns the actual audio blob instead of just a link, you need to pass the raw data of the audio file as a bytes object to the blob
parameter. Here is an example of how you can do this:
def create_audio_blob_message(audio_data: bytes, mime_type: str = 'audio/mpeg') -> ToolInvokeMessage:
"""
Create an audio blob message.
:param audio_data: The raw data of the audio file.
:param mime_type: The MIME type of the audio file.
:return: The blob message.
"""
meta = {'mime_type': mime_type}
return self.create_blob_message(blob=audio_data, meta=meta)
In this example, audio_data
should be the raw bytes of the audio file, and mime_type
should be the appropriate MIME type for the audio file (e.g., 'audio/mpeg' for MP3 files). This will ensure that the method returns the actual audio blob.
According to the Dify documentation, the create_blob_message
method is designed to handle raw data of various file types, including audio. The blob
parameter should be the raw data of the file, and the meta
parameter should include metadata such as the MIME type. This approach ensures that the method returns the actual audio blob.
def create_blob_message(self, blob: bytes, meta: dict = None, save_as: str = '') -> ToolInvokeMessage:
"""
Create a blob message.
:param blob: The blob.
:param meta: The metadata of the blob.
:param save_as: The name to save the blob as.
:return: The blob message.
"""
Additionally, you can refer to the AudioBtn
component in the Dify codebase, which demonstrates how to handle audio blobs. The loadAudio
function in the component converts the audio response data into a blob and creates an object URL for it:
const loadAudio = async () => {
const formData = new FormData()
formData.append('text', removeCodeBlocks(value))
formData.append('voice', removeCodeBlocks(voice))
if (value !== '') {
setAudioState('loading')
let url = ''
let isPublic = false
if (params.token) {
url = '/text-to-audio'
isPublic = true
}
else if (params.appId) {
if (pathname.search('explore/installed') > -1)
url = `/installed-apps/${params.appId}/text-to-audio`
else
url = `/apps/${params.appId}/text-to-audio`
}
try {
const audioResponse = await textToAudio(url, isPublic, formData)
const blob_bytes = Buffer.from(audioResponse.data, 'latin1')
const blob = new Blob([blob_bytes], { type: 'audio/wav' })
const audioUrl = URL.createObjectURL(blob)
audioRef.current!.src = audioUrl
}
catch (error) {
setAudioState('initial')
console.error('Error playing audio:', error)
}
}
}
By following this approach, you can ensure that the create_blob_message
method returns the actual audio blob instead of just a link [1][2][3][4].
To continue talking to Dosu, mention @dosu.
Self Checks
Dify version
0.6.11
Cloud or Self Hosted
Self Hosted (Source)
Steps to reproduce
I hope to return an audio clip within the built-in tool, but only a link was returned like" Link: /files/tools/b4dd3442-0f99-4072-9a65-40bd32e8bbdb.mp3". I can get audio by audio_play. The specific code is as follows: def call_tts_stream(text: str) -> Iterator[bytes]: tts_url = url tts_headers = build_tts_stream_headers() tts_body = build_tts_stream_body(text)
✔️ Expected Behavior
No response
❌ Actual Behavior
No response