Playback hitching and artifacts with StreamingSoundWave UE5.2

[madhavhaldia]

I am receiving base64 encoded mp3 audio in chunks from a websocket and trying to play this using StreamingSoundWave. During playback I am encountering hitching and artifacts. I have ruled out any network issues by implementing the same thing in python and played the audio using mpv with works great.

Everything works fine if I first append all the decoded bytes to an array and then Append to the StreamingSoundWave.

Here is my current implementation

I am calling the On Elevenlabs Websocket Message Recieved Impl method every time I receive a new audio chunk.

Here is a demo of the audio that is being played. You can hear some words being skipped and audio breaking for milliseconds in between.

https://github.com/gtreshchev/RuntimeAudioImporter/assets/60776535/8f90dc56-b54c-4b76-946c-0e7c0e284c3c

I would really appreciate if you have any insights.

[gtreshchev]

There might be three reasons for such behavior: 1) You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in 5888f50ea538fed472b5529cffbccd2a1bd25698 today to ensure sequential order based on function calls rather than decoding speed. 2) Memory reallocation might be happening very frequently, negatively impacting overall performance and causing the heard hitches. If this is the case, consider using the PreAllocateAudioData function to allocate memory once without needing to reallocate it with every append. 3) The audio playback consistently reaches its end, resulting in a brief period of empty sound before quickly appending further audio. This might be causing the overall experience of small hitches. To be honest, based on your logs, doubt that this is your issue, but it's worth at least keeping in mind.

[Zyrth]

1. You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in [5888f50](https://github.com/gtreshchev/RuntimeAudioImporter/commit/5888f50ea538fed472b5529cffbccd2a1bd25698) today to ensure sequential order based on function calls rather than decoding speed.

I ran into this same issue today. Seems like this might be the case -- Not sure if I got your update, but I solved it by making an event loop that only imports byte array data every ~0.5s. I add the chunked byte array information together using a simple append array, and then every half second the event appends the audio data and clears the array.

[madhavhaldia]

There might be three reasons for such behavior:

1. You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in 5888f50 today to ensure sequential order based on function calls rather than decoding speed.
2. Memory reallocation might be happening very frequently, negatively impacting overall performance and causing the heard hitches. If this is the case, consider using the PreAllocateAudioData function to allocate memory once without needing to reallocate it with every append.
3. The audio playback consistently reaches its end, resulting in a brief period of empty sound before quickly appending further audio. This might be causing the overall experience of small hitches. To be honest, based on your logs, doubt that this is your issue, but it's worth at least keeping in mind.

Hello, thank you so much for replying. I updated the plugin to the latest commit, unfortunately I still experience artifacts. I am pre allocating memory using the PreAllocateAudioData function.

I feel like the playback skips ahead when a new chunk is appended or something like that.

https://github.com/gtreshchev/RuntimeAudioImporter/assets/60776535/0a7be334-17cc-4853-a4a0-4ae3d412b26b

[madhavhaldia]

1. You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in [5888f50](https://github.com/gtreshchev/RuntimeAudioImporter/commit/5888f50ea538fed472b5529cffbccd2a1bd25698) today to ensure sequential order based on function calls rather than decoding speed.

I ran into this same issue today. Seems like this might be the case -- Not sure if I got your update, but I solved it by making an event loop that only imports byte array data every ~0.5s. I add the chunked byte array information together using a simple append array, and then every half second the event appends the audio data and clears the array.

Thanks a lot for the reply. Unfortunately for my use-case adding 0.5 sec latency is a no-go. I was initially appending the decoded bytes to an array and then adding that to the soundwave just like you said and It fixed the issue but I couldn't proceed with it due to the added latency.

[gtreshchev]

There might be three reasons for such behavior:

1. You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in 5888f50 today to ensure sequential order based on function calls rather than decoding speed.
2. Memory reallocation might be happening very frequently, negatively impacting overall performance and causing the heard hitches. If this is the case, consider using the PreAllocateAudioData function to allocate memory once without needing to reallocate it with every append.
3. The audio playback consistently reaches its end, resulting in a brief period of empty sound before quickly appending further audio. This might be causing the overall experience of small hitches. To be honest, based on your logs, doubt that this is your issue, but it's worth at least keeping in mind.

Hello, thank you so much for replying. I updated the plugin to the latest commit, unfortunately I still experience artifacts. I am pre allocating memory using the PreAllocateAudioData function.

I feel like the playback skips ahead when a new chunk is appended or something like that.

demo.mp4

For testing purposes, could you please initially append the audio data, then complete the appending of audio data (e.g. by adding a delay at some stage), and finally play back the sound wave after the audio data has finished appending? By following this way, you'll be able to determine whether the issue is associated with frequent memory reallocations or caused by another factor, as memory reallocations only occur during the audio data appending process for your case.

There might also be a problem with the MP3 decoder itself, which adds a slight empty sound at the beginning or end, causing such audible hitches when appending too frequently, but this needs verification. Please try the first approach described above and let me know about your results.

[madhavhaldia]

There might be three reasons for such behavior:

1. You might be appending audio data too frequently, causing inconsistent appends (e.g., when the previous audio chunk is decoded after the next one, resulting in noticeable sound shifts in small parts). If that's the case, it was fixed in 5888f50 today to ensure sequential order based on function calls rather than decoding speed.
2. Memory reallocation might be happening very frequently, negatively impacting overall performance and causing the heard hitches. If this is the case, consider using the PreAllocateAudioData function to allocate memory once without needing to reallocate it with every append.
3. The audio playback consistently reaches its end, resulting in a brief period of empty sound before quickly appending further audio. This might be causing the overall experience of small hitches. To be honest, based on your logs, doubt that this is your issue, but it's worth at least keeping in mind.

Hello, thank you so much for replying. I updated the plugin to the latest commit, unfortunately I still experience artifacts. I am pre allocating memory using the PreAllocateAudioData function. I feel like the playback skips ahead when a new chunk is appended or something like that. demo.mp4

For testing purposes, could you please initially append the audio data, then complete the appending of audio data (e.g. by adding a delay at some stage), and finally play back the sound wave after the audio data has finished appending? By following this way, you'll be able to determine whether the issue is associated with frequent memory reallocations or caused by another factor, as memory reallocations only occur during the audio data appending process for your case.

There might also be a problem with the MP3 decoder itself, which adds a slight empty sound at the beginning or end, causing such audible hitches when appending too frequently, but this needs verification. Please try the first approach described above and let me know about your results.

Thank you so much for your continued support. Appending the audio data until all the data has finished appending and then calling play doesn't fix this issue.

Collecting all the audio data in an array and then appending all that data in one go fixes this issue.

Edit: If I perform process one with PCM data, the issue is fixed so I am inclined to believe this is an issue with mp3 decoding as well.

Edit2: I replaced dr_mp3 with minimp3 and the issue still persists so not a dr_mp3 issue. I read that dr_mp3 is based off minimp3 though so if the issue is with minimp3 then it might propagate to dr_mp3.

[Zyrth]

I can confirm on my end that using append from raw on a s16 / 24000 PCM works in the same scenario mp3 has the artifacts

[madhavhaldia]

Hey! Figured this is an issue with minimp3. For my particular use case it required at least 416 bytes of audio to be present to decode properly. The decoder is also reinitialised for every chunk of audio appended which also has some overhead. For this niche use case I have implemented a streaming system on my own.

@gtreshchev Thanks a lot for the support and for creating this awesome plugin!

[mxbi]

@madhavhaldia Is there any chance you're willing to share your solution to this in more detail? I'm running into the same issue in a very similar scenario, and it would be awesome to see how you implemented your streaming system.