Flaky test: testTwoBPInstancesWithVideo

MobileNativeFoundation / bluepill

Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine

BSD 2-Clause "Simplified" License

3.19k stars 232 forks source link

Flaky test: testTwoBPInstancesWithVideo #469

Open chenxiao0228 opened 3 years ago

chenxiao0228 commented 3 years ago

testTwoBPInstancesWithVideo has been very flaky in the checks.

I briefly looked into it and have these findings

I could not reproduce it on my machine
tried to change it to 1 simulator, still flaky
tried to add some debug logging in the code, could not get too much info. Next step could be piping the output of the process to a file and capture it.
recording videos uses xcrun simctl io, which could be pretty expensive

For now I think we should disable it and try to root cause it in a branch.

chenxiao0228 commented 3 years ago

@RainNapper FYI since you worked on this feature.

RainNapper commented 3 years ago

I think disabling it sounds good. I thought it was fixed in XC12, but it looks to have gone back. This has been working in production, so I'm not sure what's different about the test configuration.

On Tue, Jan 19, 2021 at 12:07 PM Xiao Chen notifications@github.com wrote:

@RainNapper https://github.com/RainNapper FYI since you worked on this feature.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/linkedin/bluepill/issues/469#issuecomment-763096873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALB5OE4524Y6QW5EKINTXDS2XQ7ZANCNFSM4WJK2PKQ .

ob commented 3 years ago

I wonder if it's just memory on the macOS VMs or some other resource that gets consumed... we could try running ps and vm_stat in a loop in a separate process to see how the machine is doing...