search

mongodb / stitch-ios-sdk

Apache License 2.0
42 stars 25 forks source link

STITCH-2654: Swift performance testing harness #158

Closed tkaye407 closed 5 years ago

tkaye407 commented 5 years ago

I left some of the logging in here in case I need to debug and because I assume I will have to make a few changes after you all look at it. For now, a few things to note

  1. The networkMonitor and transport should be fully functional for both prod and local
  2. Prod tests still need to have the Stitch Server running in the background. Although it doesn't do anything with it, it seems to randomly fail with no explanation when the stitch server is not running. I was not sure how important this is though.
  3. We now have the ThrowingCallBackJoiner which on call to value() will throw if there was an error. This allows us to catch errors in the harness
  4. Errors now update the test document to have a status of "failed" and adds a description of the failure.
  5. I haven't seen them in a while, but there appears to be some flakey-ness in the setup/teardown logic of local tests where it occasionally fails. I haven't seen it in the last 20 or so runs, but I would appreciate it if you take special note of the setup/teardown logic as you go through this PR.
tkaye407 commented 5 years ago

Made a bunch of changes and tested this out a lot. The only problem I have found is that when running tests against prod the instrumented network transport only works for the first test run, which is a problem that ill continue to look into; however, I am not currently seeing the problem