search

cashapp / AccessibilitySnapshot

Easy regression testing for iOS accessibility
Apache License 2.0
541 stars 69 forks source link

Debug flaky parser tests #36

Closed NickEntin closed 3 years ago

NickEntin commented 3 years ago

Adds a bunch of logs to help debug #29

NickEntin commented 3 years ago

Hmm, 4 builds in a row have passed now. 🤔 Something in the debugging code resolved the issue.

NickEntin commented 3 years ago

So it definitely seems like the log statements in the parser are what's fixing the build. Which... definitely shouldn't fix the build. Just to make sure there's nothing else going on...

NickEntin commented 3 years ago

So even something as simple as adding a test attachment is causing the build to be stable. I'm thinking this must be a very slight timing issue, although I can't think of why the timing would matter here.

NickEntin commented 3 years ago

Testing out replacing the print statement with a call to fflush, per @efirestone 's suggestion.

Running tally: Passed Failed
4 0
NickEntin commented 3 years ago

Testing with a totally empty print statement

Running tally: Passed Failed
4 0
NickEntin commented 3 years ago

Trying something that should definitely be a no-op

Running tally: Passed Failed
4 0
NickEntin commented 3 years ago

Original code, different SHA

Running tally: Passed Failed
1 1
NickEntin commented 3 years ago

Checking whether changing something in the test target will fix this, meaning that the problem is in the test target rather than the framework itself

Running tally: Passed Failed
8 0
NickEntin commented 3 years ago

Since a change in the test target seems to make the build stable, it feels like the problem is actually in the test target and not the framework itself.