subliminal-test can hang if Instruments puts up an alert at the end of the tests

[aegolden]

I've seen Jenkins CI jobs hang a couple of times in the subliminal-test script, when all the tests for the job have completed but Instruments shows an alert like "The simulated application quit."

I think the launch_instruments function in subliminal-test is waiting for Instruments to exit, and Instruments doesn't exit if it shows an alert like that.

[wearhere]

Hm. We could make an applescript to watch for and dismiss such an alert, similar to the script that subliminal-test uses to watch for and dismiss instruments' authorization alert.

[justinseanmartin]

"Simulator session timed out" is the error that we're seeing. We're not sure of the condition that is causing this to happen, but recently it started happening much more frequently (at least 2-3 times a day). Unfortunately, as far as we can tell, AppleScript appears to be unable to enumerate the alert window for the "instruments" process.

    get name of item 19 of every process
        --> "instruments"
    (*instruments*)
    count every window of item 19 of every process
        --> 0
    get entire contents of item 19 of every process
        --> {}

And here is our script:

tell application "System Events"
    repeat with theProcess in (every process)
        tell theProcess
            log (name as string)
            repeat with theWindow in windows
                get entire contents of theWindow
            end repeat
            get entire contents of theProcess
        end tell
    end repeat
end tell

I really haven't worked in AppleScript much at all, so it is very possible that we're just doing something horribly wrong in the script.

[wearhere]

We believe(d, at least at one time) that the reason for the error @aegolden saw was because we exit() from Subliminal's completion handler rather than letting Instruments kill the app. @justinseanmartin says (in chat) that he's not doing that, yet still seeing this error... they still sound pretty similar to me.

You can see the script we were using to catch alerts here. I'm not sure why that would work and the script above wouldn't. But @justinseanmartin has an even nicer way to fix this sort of thing, which is to load a dylib into instruments to prevent it from showing NSAlerts. I think we should do that–subliminal-instrument is in the perfect position to load such a dylib, and this approach will give us a chance to examine the alert and perhaps pass certain errors through to subliminal-instrument (which I think we should do).

[wearhere]

I don't think that suppressing the alert is even that hacky. @justinseanmartin points out that there's only 1 button to push and no input required–so us suppressing the alert is equivalent to it having shown and been dismissed.

[justinseanmartin]

Here is my WIP branch:

https://github.com/justinseanmartin/Subliminal/tree/jmartin/issue-97

I'm not certain that this is working as I expect it too, but I'll be monitoring it over the next few days.