search

soundscape-community / soundscape

An iOS application/service that aids navigation through spatialized audio
https://soundscape.services
MIT License
20 stars 19 forks source link

Random test failures #106

Open RDMurray opened 2 weeks ago

RDMurray commented 2 weeks ago

Sometimes the tests in GitHub are failing and I don't know why. The latest run failed but then succeeded when I re-ran it.

It does seem to be running the tests on iOS 18.1 which I think is still in beta. Perhaps we should force it to use 18.0 or the latest version of iOS 17 for now?

steinbro commented 2 weeks ago

I ran into this too. I agree enforcing a stable iOS version is a good thing to try. I had also noticed that some of the versions of the reusable GitHub Action stages we leverage were reported as deprecated, but it looks like you already fixed that.

Separate issue, but I also noticed that the pipeline isn't accurately noticing when the build stage fails. This makes the output confusing, because the pipeline carries on to the subsequent test stages which complain about the app bundle not existing. Example: https://github.com/soundscape-community/soundscape/actions/runs/10754605735/job/29825643227

RDMurray commented 1 week ago

set to iOS 18.0 in #107

The problem was that piping the build output into xcpretty swallowed the exit code. that should also be fixed in #107 according to xcpretty documentation.

steinbro commented 1 week ago

It looks like this is still a problem (the most recent commit still needed to be run twice before it passed). I dug into the output log and have a theory, though not a fix.

A common property of the inconsistently failing tests is triggering speech. Based on some discussions online (e.g. https://stackoverflow.com/a/73886077, https://forums.developer.apple.com/forums/thread/708114), as of iOS 16, the simulator no longer comes with voices installed by default -- you need to manually open the Settings app within the simulator and install voices individually. This is consistent with what I observed after an upgrade some time within the last year, when all the voices disappeared from inside the simulator on my Mac. I'm not sure how/if this can be done in an automated way inside a pipeline. Nor can I explain why the failures are only inconsistent.

Below is an excerpt of the output from a failing test:

Test Case '-[UnitTests.AudioEngineTest testDiscreteAudio2DSimple]' started.
2024-09-20 17:26:14:571 [AUS] Category set (playback, default, options: none)
2024-09-20 17:26:14:571 [AUD] Initializing audio engine
2024-09-20 17:26:14:572 [AUD] Connecting audio graph in 3D mode (from <OperationQueue: NSOperationQueue Main Queue>)
2024-09-20 17:26:14:578 [AUS] Audio session activated
2024-09-20 17:26:14:578 [AUD] Starting audio engine (from services.soundscape.audioengine)
2024-09-20 17:26:14:591 [AUD] Resuming 0 players
2024-09-20 17:26:14:591 [AUD] Audio engine does not need to be started (it is already running)
2024-09-20 17:26:14:591 [AUD] Resuming 0 players
2024-09-20 17:26:14.604324+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/TextToSpeech.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:14.639504+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/TextToSpeech.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:14.651305+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/AccessibilityUtilities.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:15:436 [AUD] Tried to play next sound when the current sound was still playing (Optional(C4ED4483-A943-44F2-BC9F-200D513CAF31))...
2024-09-20 17:26:15:437 [AUD] Tried to play next sound when the current sound was still playing (Optional(C4ED4483-A943-44F2-BC9F-200D513CAF31))...
2024-09-20 17:26:16.354521+0000 Soundscape[13330:54581] [asset] Failed to get sandbox extensions
2024-09-20 17:26:16.438687+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.441707+0000 Soundscape[13330:54581] [asset] #FactoryInstall Unable to query results, error: 5
2024-09-20 17:26:16.445212+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.475853+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.476080+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.478170+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.478843+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.473348+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.489509+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.587649+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.589387+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.596751+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.597445+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.780426+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.782910+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.797422+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.802185+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.825969+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.827267+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:24.398982+0000 Soundscape[13330:54468] [AMCP]  50955          HALC_ProxyIOContext.cpp:1619  HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
/Users/runner/work/soundscape/soundscape/apps/ios/UnitTests/Audio/AudioEngineTest.swift:64: error: -[UnitTests.AudioEngineTest testDiscreteAudio2DSimple] : XCTAssertEqual failed: ("XCTWaiterResult(rawValue: 2)") is not equal to ("XCTWaiterResult(rawValue: 1)")
/Users/runner/work/soundscape/soundscape/apps/ios/UnitTests/Audio/AudioEngineTest.swift:65: error: -[UnitTests.AudioEngineTest testDiscreteAudio2DSimple] : XCTAssertEqual failed: ("0") is not equal to ("1")
Test Case '-[UnitTests.AudioEngineTest testDiscreteAudio2DSimple]' failed (24.770 seconds)
RDMurray commented 1 week ago

Apparently using the tts api triggers a voice to be installed. I suspect it succeeds the second time because GitHub is caching the macos instance and the voice has already been downloaded

We could possibly add a test that runs first to trigger a voice download and waits for it to be availible.