search

soundscape-community / soundscape

An iOS application/service that aids navigation through spatialized audio
https://soundscape.services
MIT License
21 stars 22 forks source link

Random test failures #106

Open RDMurray opened 2 months ago

RDMurray commented 2 months ago

Sometimes the tests in GitHub are failing and I don't know why. The latest run failed but then succeeded when I re-ran it.

It does seem to be running the tests on iOS 18.1 which I think is still in beta. Perhaps we should force it to use 18.0 or the latest version of iOS 17 for now?

steinbro commented 2 months ago

I ran into this too. I agree enforcing a stable iOS version is a good thing to try. I had also noticed that some of the versions of the reusable GitHub Action stages we leverage were reported as deprecated, but it looks like you already fixed that.

Separate issue, but I also noticed that the pipeline isn't accurately noticing when the build stage fails. This makes the output confusing, because the pipeline carries on to the subsequent test stages which complain about the app bundle not existing. Example: https://github.com/soundscape-community/soundscape/actions/runs/10754605735/job/29825643227

RDMurray commented 2 months ago

set to iOS 18.0 in #107

The problem was that piping the build output into xcpretty swallowed the exit code. that should also be fixed in #107 according to xcpretty documentation.

steinbro commented 2 months ago

It looks like this is still a problem (the most recent commit still needed to be run twice before it passed). I dug into the output log and have a theory, though not a fix.

A common property of the inconsistently failing tests is triggering speech. Based on some discussions online (e.g. https://stackoverflow.com/a/73886077, https://forums.developer.apple.com/forums/thread/708114), as of iOS 16, the simulator no longer comes with voices installed by default -- you need to manually open the Settings app within the simulator and install voices individually. This is consistent with what I observed after an upgrade some time within the last year, when all the voices disappeared from inside the simulator on my Mac. I'm not sure how/if this can be done in an automated way inside a pipeline. Nor can I explain why the failures are only inconsistent.

Below is an excerpt of the output from a failing test:

Test Case '-[UnitTests.AudioEngineTest testDiscreteAudio2DSimple]' started.
2024-09-20 17:26:14:571 [AUS] Category set (playback, default, options: none)
2024-09-20 17:26:14:571 [AUD] Initializing audio engine
2024-09-20 17:26:14:572 [AUD] Connecting audio graph in 3D mode (from <OperationQueue: NSOperationQueue Main Queue>)
2024-09-20 17:26:14:578 [AUS] Audio session activated
2024-09-20 17:26:14:578 [AUD] Starting audio engine (from services.soundscape.audioengine)
2024-09-20 17:26:14:591 [AUD] Resuming 0 players
2024-09-20 17:26:14:591 [AUD] Audio engine does not need to be started (it is already running)
2024-09-20 17:26:14:591 [AUD] Resuming 0 players
2024-09-20 17:26:14.604324+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/TextToSpeech.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:14.639504+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/TextToSpeech.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:14.651305+0000 Soundscape[13330:54459] [general] NSBundle file:///Library/Developer/CoreSimulator/Volumes/iOS_22A3351/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS%2018.0.simruntime/Contents/Resources/RuntimeRoot/System/Library/PrivateFrameworks/AccessibilityUtilities.framework/ principal class is nil because all fallbacks have failed
2024-09-20 17:26:15:436 [AUD] Tried to play next sound when the current sound was still playing (Optional(C4ED4483-A943-44F2-BC9F-200D513CAF31))...
2024-09-20 17:26:15:437 [AUD] Tried to play next sound when the current sound was still playing (Optional(C4ED4483-A943-44F2-BC9F-200D513CAF31))...
2024-09-20 17:26:16.354521+0000 Soundscape[13330:54581] [asset] Failed to get sandbox extensions
2024-09-20 17:26:16.438687+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.441707+0000 Soundscape[13330:54581] [asset] #FactoryInstall Unable to query results, error: 5
2024-09-20 17:26:16.445212+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.475853+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.476080+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.478170+0000 Soundscape[13330:54581] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.478843+0000 Soundscape[13330:54581] [catalog] Unable to list voice folder
2024-09-20 17:26:16.473348+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.489509+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.587649+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.589387+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.596751+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.597445+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.780426+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServicesVocalizerVoice failed: 2
2024-09-20 17:26:16.782910+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.797422+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.GryphonVoice failed: 2
2024-09-20 17:26:16.802185+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:16.825969+0000 Soundscape[13330:53937] [catalog] Query for com.apple.MobileAsset.VoiceServices.CustomVoice failed: 2
2024-09-20 17:26:16.827267+0000 Soundscape[13330:53937] [catalog] Unable to list voice folder
2024-09-20 17:26:24.398982+0000 Soundscape[13330:54468] [AMCP]  50955          HALC_ProxyIOContext.cpp:1619  HALC_ProxyIOContext::IOWorkLoop: skipping cycle due to overload
/Users/runner/work/soundscape/soundscape/apps/ios/UnitTests/Audio/AudioEngineTest.swift:64: error: -[UnitTests.AudioEngineTest testDiscreteAudio2DSimple] : XCTAssertEqual failed: ("XCTWaiterResult(rawValue: 2)") is not equal to ("XCTWaiterResult(rawValue: 1)")
/Users/runner/work/soundscape/soundscape/apps/ios/UnitTests/Audio/AudioEngineTest.swift:65: error: -[UnitTests.AudioEngineTest testDiscreteAudio2DSimple] : XCTAssertEqual failed: ("0") is not equal to ("1")
Test Case '-[UnitTests.AudioEngineTest testDiscreteAudio2DSimple]' failed (24.770 seconds)
RDMurray commented 2 months ago

Apparently using the tts api triggers a voice to be installed. I suspect it succeeds the second time because GitHub is caching the macos instance and the voice has already been downloaded

We could possibly add a test that runs first to trigger a voice download and waits for it to be availible.

RDMurray commented 2 weeks ago

Unrelated to the voice issue, but it is failing again with the latest macos image on github actions. It worked with version 20241022.361, but failed with version 20241108.442.

It doesn't seem to be accepting the target specification:

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:arm64, variant:Designed for [iPad,iPhone], id:0000FE00-C561D0E269705BEC, name:My Mac }
(...}

According to the readme for the image the iOS 18 simulator is only available with Xcode 16, and 18.1 with Xcode 16.1.

I'm not sure there's much we can do other than to manually update the workflow each time a version becomes unsupported.

RDMurray commented 2 weeks ago

Seems to be okay after updating the workflow to iOS 18.1 / Xcode 16.1

steinbro commented 2 weeks ago

Should we revert to the action using the latest, possibly beta iOS version? Hard-coding the iOS version didn't fix the problem here, and I'd rather have potential future failures reported earlier than be stuck on a single version until someone manually changes it.

RDMurray commented 1 week ago

There's the Xcode version to deal with as well. I have it working with the latest stable Xcode, iOS version determined automatically by Xcode in my fork.

The run is here: https://github.com/RDMurray/soundscape/actions/runs/11818261318/job/32925422972#step:3:1