Crashing randomly occurring if using a task inside another async class without storing the reference + Crashes occurring when trying to use null task in logs details

[tumi23]

Hey,

The plugin seems to crash in our project at random when it finishes generating an audio, this seems to be two different crashes as far as I know. One is related to the line at 58 in AzSpeechRunnableBase.cpp ->"FScopeLock Lock(&OwningTask->Mutex);" and happens both in editor and in a packaged game, and here is the crash log.

LoginId:00b08ca546492448f7134499d71f6560
EpicAccountId:a6049f67e5b84e6dac3608799f140590

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION writing address 0x0000000000000024

ntdll
ntdll
ntdll
UnrealEditor_AzSpeech!FAzSpeechRunnableBase::Exit() [C:\Users\XXX\wkspaces\XXX\Plugins\AzSpeech_UE51\Source\AzSpeech\Private\AzSpeech\Runnables\Bases\AzSpeechRunnableBase.cpp:58]
UnrealEditor_AzSpeech!FAzSpeechSynthesisRunnable::Exit() [C:\Users\XXX\wkspaces\XXX\Plugins\AzSpeech_UE51\Source\AzSpeech\Private\AzSpeech\Runnables\AzSpeechSynthesisRunnable.cpp:78]
UnrealEditor_Core!FRunnableThreadWin::Run() [D:\build++UE5\Sync\Engine\Source\Runtime\Core\Private\Windows\WindowsRunnableThread.cpp:152]

and the other one I've only seen when we package the project and is related to calling the destructor of AzSpeechRunnableBase derived classes. This one is harder to track down because of the debug symbols not being packaged with an epic launcher version of the engine, but might be related to the other crash because when I attach an IDE debugger to it I can see the name of the functions that are being called. The functions being called are the destructors of the AzSpeechRunnableBase class that is being destroyed. Here are the last lines of the log file for when this crashes as the call stack isn't much use without symbols.

[2023.01.11-16.35.11:615][455]LogAzSpeech: Display: Task: TextToSoundWave (68130); Function: SetReadyToDestroy; Message: Setting task as Ready to Destroy
[2023.01.11-16.35.11:615][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68130); Function: StopAzSpeechRunnableTask; Message: Setting runnable work as pending stop
[2023.01.11-16.35.11:655][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ProcessSynthesisResult; Message: Task running. Reason: SynthesizingAudio
[2023.01.11-16.35.11:656][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ProcessSynthesisResult; Message: Task running. Reason: SynthesizingAudio
[2023.01.11-16.35.11:707][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ProcessSynthesisResult; Message: Task running. Reason: SynthesizingAudio
[2023.01.11-16.35.11:707][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ProcessSynthesisResult; Message: Task running. Reason: SynthesizingAudio
[2023.01.11-16.35.11:708][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ProcessSynthesisResult; Message: Task completed. Reason: SynthesizingAudioCompleted
[2023.01.11-16.35.11:708][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: StopAzSpeechRunnableTask; Message: Setting runnable work as pending stop
[2023.01.11-16.35.11:782][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: ClearSignals; Message: Disconnecting Azure SDK recognition signals
[2023.01.11-16.35.11:782][455]LogAzSpeech_Internal: Display: Task: TextToSoundWave (68129); Function: RemoveBindings; Message: Removing existing delegate bindings

[lucoiso]

@tumi23 created a new branch and made adjustments to avoid the usage of invalid owning tasks: https://github.com/lucoiso/UEAzSpeech/tree/hotfix/CRASH-RUNNABLE-68

[tumi23]

Thank you! I'll try it out and see if it still crashes

[tumi23]

@lucoiso Still crashing it seems, here is a picture from the debugger because the engine didn't generate a crash report with the engine open through an IDE

[lucoiso]

added a mutex to the runnable thread and set to lock the scope in the beginning of exit func @tumi23 👀

[tumi23]

Sick, i'll update the code and see if it still occurs

[tumi23]

Crashed again and got the azspeech log file as well as the final lines in the log file related to the crash.

Another odd thing to add from the video I got sent from Q&A of the crash is that the game continues running and the dialogue does actually finishes generating and starts playing correctly but once you click ok on the crash message it ends execution and closes.

CrashLog.txt UEAzSpeech 2023.01.23-11.48.58.log

[lucoiso]

@tumi23

Could you provide an example of implementation with this issue happening? I've been trying to help resolve it, but I still haven't been able to replicate this error here.

I'm currently testing with both Text to Sound Wave and SSML to Sound Wave, but with no success in replicating the issue.

Being able to reproduce the issue here will make it easier to identify and resolve. It's very strange what's happening, I hope to resolve this soon. 😵‍💫

[tumi23]

Let me see if I can find some way to replicate it, it has been extremely random so far whether it happens or not

[tumi23]

I think I may have figured out the problem and it might be due to my own stupidity, and I'm not so sure its anything you need to fix on your end. It seems that I was creating the Asynchronous task for generating text to speech but forgot to keep the reference to them so inevitably the garbage collector would swoop in and destroy the task while it was most likely in the middle of generating or execution some thread sensitive code which then triggers the ScopeLock crash. If its something you want to look into you can in C++(at least this is what my code is doing) create an asynchronous task class that then creates a UTextToSoundWaveAsync task but doesn't store the reference to the task itself.

At the moment I'm just guessing with this but it seems highly likely that this is why it crashes randomly only for me.

[lucoiso]

@tumi23

Sorry for the delay, made a new change:

• Register all tasks with the game instance via:

  /**
   * Call to globally register this object with a game instance, it will not be destroyed until SetReadyToDestroy is called
   * This allows having an action stay alive until SetReadyToDestroy is manually called, allowing it to be used inside loops or if the calling BP goes away
   */
  virtual void RegisterWithGameInstance(const UObject* WorldContextObject);

👀

[lucoiso]

Merged in development branch: https://github.com/lucoiso/UEAzSpeech/tree/development

Feel free to contact if the problem persists and I'll create a new issue/branch focused on the persistent problem! I'm already merging because of the first changes related to logging, etc! : )

[tumi23]

Will do, we ended up needing to disable the text to speech. I'll need turn it back on soon and see if it still causes problem.