search

braintree / braintree_ios

Braintree SDK for iOS
https://developer.paypal.com/braintree/docs/start/hello-client/ios/v5
MIT License
556 stars 290 forks source link

Test Venmo UI tests in CI/CD #1328

Open KunJeongPark opened 1 month ago

KunJeongPark commented 1 month ago

Summary of changes

Locally, these tests were passing most of the time - I had 2 failures on 2 or three tests on simulator with iOS 17.0.1

On pushing to remote branch, there were failures on all my trials except for #3 and on second run of the same change, I had two failures:

  1. Adding teardown function
  2. iPhone 15 simulator of tests.yml
  3. Try iPhone 15 simulator with iIS 17.4
  4. Try Xcode 15.4 + simulator iPhone 15 and iOS 17.2
  5. Try Xcode 15.0.1 and OS 17.0.1 to see if issue is compatibility of Xcode and OS

They all had some failures in the Venmo UI tests, except for one run with iPhone 15 simulator with iOS 17.2 (no failures on any UITests)

There seems to be some improvement with use of iPhone 15 simulator on tests.yml file.

Checklist

Authors

List GitHub usernames for everyone who contributed to this pull request.

jaxdesmarais commented 2 weeks ago

@KunJeongPark it looks like the UI tests are still failing

KunJeongPark commented 2 weeks ago

@KunJeongPark it looks like the UI tests are still failing

Yes, I couldn't get rid of all failures. I noted that using iPhone 15 simulator in tests.yml seemed to reduce the number of failures.

I listed all the things I've tried in the description. Sarah wanted me to time box it and close it out, so I thought I'd keep using iPhone 15 simulator which seemed to reduce the number of failures.

jaxdesmarais commented 1 week ago

Yes, I couldn't get rid of all failures. I noted that using iPhone 15 simulator in tests.yml seemed to reduce the number of failures. I listed all the things I've tried in the description. Sarah wanted me to time box it and close it out, so I thought I'd keep using iPhone 15 simulator which seemed to reduce the number of failures.

I totally understand timeboxing the work. I applied the changes from this PR as well as what looks to be a spike Rich started a while ago with some other changes from some research online in PR #1345. I'll work on cleaning that PR up further when I get back from OOO so we can hopefully get our UI Tests back in the green to make our CI more reliable!