Open jesmrec opened 6 years ago
Here it is a little summary of the discussions/thoughts @felix-schwarz and I had about how to make UI testing in the app more straightforward:
There is a mechanism called Host Simulator in the SDK that intercepts previously selected network calls and mock the response for this particular call with a set response. There are examples of how to use this mechanism in some SDK tests, like here
The problem using the Host Simulator is that is going to be impossible to test particular app states in a scalable way.
Set the values of the SDK objects directly to that they need to be to test the app. For read-only properties or private variables, we can leverage the -setValue:forKey:
method from KVO.
Swizzle individual methods to replace them with versions to return the values needed for testing.
Create a modified version of the SDK, taking advantage of the C preprocessor, like so:
// ModifiedOCQuery.m
// Turn OCQuery into BaseOCQuery
#define OCQuery BaseOCQuery
#import "OCQuery.m"
#undef OCQuery
// Subclass BaseOCQuery
@interface OCQuery : BaseOCQuery
@end
@implementation OCQuery
- (void)methodToOverride;
@end
Another thing that came up to my mind is to have a private ownCloud server only set up for app testing and, through some script, or OCcomands, leave the server in the state we want for the whole app testing or even for each specific test-suit.
These ideas are the result of some light discussions, there could be more suitable options than what is summarized in this comment. Also if I miss something or I've written something wrong @felix-schwarz you can correct me.
the options that seem to be more suitable are KVO and modified SDK, but both cases will need to develop testable code and estimate efforts.
The case of private oC server fits for the case of E2E testing, in which we are testing app + sdk, where the sdk is already covered with unit tests. We also lose the perspective of the app testing, so one failure can be caused in the sdk making wrong positives in the app.
After check the options and see examples of swift projects provided by @pablocarmu (thanks!) IMHO the best solution it is:
preprocessor macro
on the ios-app to call the right classes (test subclasses or sdk classes, check the code at the end of the comment)if @jesmrec @pablocarmu and @felix-schwarz agree with this implementation I can start with it 👍
#if os(macOS)
import AppKit
public typealias View = NSView
public typealias Color = NSColor
public typealias Image = NSImage
public typealias Screen = NSScreen
public typealias Font = NSFont
public typealias EdgeInsets = NSEdgeInsets
internal typealias ClickGestureRecognizer = NSClickGestureRecognizer
#else
import UIKit
public typealias View = UIView
public typealias Color = UIColor
public typealias Image = UIImage
public typealias Screen = UIScreen
public typealias Font = UIFont
public typealias EdgeInsets = UIEdgeInsets
internal typealias ClickGestureRecognizer = UITapGestureRecognizer
#endif
Here's a solution that enables mocking, but requires no changes to existing code (which is something I'd like to avoid TBH):
Create a new build configuration: ios-sdk Create a new build configuration "Mock" in the SDK that includes a prefix header that changes all names per #define:
#define OCCore MOCCore
#define OCQuery MOCQuery
That makes sure that the classes are actually called MOCCore
and MOCQuery
in the binary.
Create a new build configuration: ios-app Create a new build configuration "Mock" in the app. This makes sure Xcode builds and links the "Mock" version of the SDK.
Also, add a define in the build settings for this new configuration MOCK=1
.
Add type aliases or mocking subclasses to ios-app In the ios-app, it's then possible to do this:
#if MOCK
typealias OCCore = MOCCore
class OCQuery : MOCQuery {
// Subclass the methods you want to mock
}
#endif
Change ios-app project settings to use the Mock build configuration for testing To make sure, the mocking version is always used during testing, change the scheme in Xcode so that it uses the Mock build configuration.
--
Please let me know if you agree @pablocarmu @jesmrec @javiergonzper. I can make these changes relatively quickly then.
@felix-schwarz for me it is ok 👍 it should works
WFM
I just tried to implement my suggestion but had to realize typealias
and #define
s won't do:
MOCCore
would still be creating and returning instances of MOCItem
s, even if the app subclasses MOCItem
as OCItem
. So the reach of subclassing here is generally limited to object the app creates itself. Which are almost none. Which disqualifies subclassing for mocking the SDK IMO.
Even leaving that aside, Swift insists on type safety. There's no problem as long as a type is just typealias
ed, but as soon as a type is subclasses to its "original" name, Swift throws Cannot assign value of type 'MOCItem?' to type 'OCItem?'
-like compilation errors all over the code base.
serialization and deserialization of existing data creates errors as it can't find the classes that are merely typealiased
Ok, so with that approach (and typealiasing in general - for this purpose) off the table, what else could be done? I see three options:
1) insert mocking code directly into the SDK, enabled/disabled with #ifdefs. I would like to avoid this.
2) keep a "Mock" target and within that:
OCCore.startQuery
) to a block (i.e. (core: OCCore, query: OCQuery) -> Void
)3) keep a "Mock" target and within that:
extension
s that implement special mocking versions of existing methods, i.e. func mock_startQuery(query: OCQuery)
.Personally, I'd avoid 1), consider 3), but likely go for 2) because it gives the most flexibility and possibly the cleanest code.
@javiergonzper @jesmrec @pablocarmu
The feature/sync
branch of the SDK adds the new ownCloudMocking.framework
.
Linking against it provides access to OCMockManager
, which implements what I've been writing about above.
Check out https://github.com/owncloud/ios-sdk/blob/137792e14b888de51cb0eef45de7684d2f9a8f41/ownCloudMockingTests/ownCloudMockingTests.m for a simple example of how mocking works for this class: https://github.com/owncloud/ios-sdk/blob/137792e14b888de51cb0eef45de7684d2f9a8f41/ownCloudMockingTests/OCMockTestClass.h .
I created the branch feature/mockTestFromSDK
on the app and on the SDK with the code to make a an example test with mocking using swizzling. It is working 👍.
Right now it could be an entry point to start mocking the UI tests mocking the necessary methods of the SDK to all the tests.
I am happy to say that we have our first UI Tests on the app that mock the response of the server.
Right now we can develop the next test using as example the mock of the class OCConnection
and the method prepareForSetup
.
I created a couple of PR one for the app and other for the SDK:
ios-app: https://github.com/owncloud/ios-app/pull/136
ios-sdk: https://github.com/owncloud/ios-sdk/pull/30
Before merge the PR of the ios-app we need to fix the compilation issues on the branch feature/fileprovider
After develop several tests we are discovering the limitations of EarlGrey:
The app is not rebooted after each test This is a problem because every test must return the app to the original View. If the test fails the view will not be restored to the original view so all the following tests probably will fail.
It is not possible to access to the real objects in the screen For example to get the number of cells or access a concrete cell in a table to get the title value.
I would recommend to start using Kif (https://github.com/kif-framework/KIF) and remove EarlGrey from the project.
If no one have any inconvenience about it I want to start to do the change as soon as possible CC: @jesmrec @michaelstingl @mneuwert @felix-schwarz @pablocarmu
The app is not rebooted after each test This is a problem because every test must return the app to the original View. If the test fails the view will not be restored to the original view so all the following tests probably will fail.
This is really scary. Each test must be independent from all other tests, and should start from a clean status of the app, not depending on the way they are executed. If tearDown
does not reset completely the app, we are on the risk of having "binary" tests: tests that tell you either "all works" or "something does not work, but i'm not going to tell you what o where the it is". Of course, this is better than not having tests, but in case of having a huge amount of tests, it will not help.
Does KIF work in that way?
The EarlGrey FAQ explains:
EarlGrey is unable to launch or terminate the app under test from within the test case, something that Xcode UI Testing is capable of.
To me it seems best practice to navigate to a specific spot in the UI - and back out, to verify navigation works both ways. Navigation to a specific point in the app, and back to the "start screen", should be put in reusable methods anyway to keep the tests maintainable, so a typical test could look like this (pseudo-code):
func testSomething() {
self.createBookmark()
self.loginToBookmark(atPosition: 0)
// UI test
self.logout()
self.deleteLastBookmark()
}
Bonus idea: "building blocks" like loginToBookmark
could automatically push blocks of code to an array, which can then be executed in reverse order to go back to the "start screen". The code could then be simplified like this:
func testSomething() {
self.createBookmark() // adds block with self.deleteLastBookmark() to array
self.loginToBookmark(atPosition: 0) // adds block with self.logout() to array
// UI test
self.resetUI() // runs blocks in reverse order (might even move this into tearDown()).
}
WKWebView
vs Aug, 23rd, adding code to avoid XCUIScreenshot
in Xcode 10 beta(!)) – and it's created, backed and used by Google for their own apps.I see no reason or advantage to switch to KIF and strongly recommend staying with EarlGrey.
CC: @javiergonzper @jesmrec @michaelstingl @mneuwert @pablocarmu
Unlike Xcode UI tests, both KIF and EarlGrey base on XCTest, run inside the app itself and therefore neither KIF nor EarlGrey can relaunch the app.
To me it seems best practice to navigate to a specific spot in the UI - and back out, to verify navigation works both ways. Navigation to a specific point in the app, and back to the "start screen", should be put in reusable methods anyway to keep the tests maintainable, so a typical test could look like this (pseudo-code):
more that relaunching the app, we need to launch every test from scratch. I guess KIF allows to restart the app (not reinstalling), so every test will start from the same point, regardless of the status in which the previous test finished. This is that Earlgrey lacks, and make all test fails after the first failure.
From QA pov, each test should be as much scoped as posible. Browsing through views before/after the main actions will increase the risk of failures that are no caused by the target of the test, and then, not-desirable false positives.
In the event that one test fails and other, subsequent tests fail because of that, we can see in the logs which test failed first and for what reason. We can then work to fix the issue, so the test no longer fails. If fixing the test takes longer, we can temporarily rename it to start with something other than "test" so Xcode no longer executes it.
i think that tests are developed to avoid such things. One test is intended to cover one scenario. When the test fail, you know that something is broken in the scenario and then, research about it to get the bug or whatever.
Discussing about EarlGrey
or KIF
(or other one) is the best, does not have place. We should choose the best for our necessities.
I will not deep dive in the technical stuff, so i trust in all of you, but what i wrote here https://github.com/owncloud/ios-app/issues/58#issuecomment-439016099 needs to be studied because is one of the first premises to get good and optimal tests.
more that relaunching the app, we need to launch every test from scratch. I guess KIF allows to restart the app (not reinstalling),
I don't see how KIF should pull that off, as it runs inside the app itself. As soon as it terminates the app, it terminates itself and can no longer execute code.
The KIF developers say pretty much the same:
KIF runs in a unit test target, not a UI test. If one test fails, the app will likely not be in a testable state at that point, and it makes sense for the tests to stop.
You can't fully restart the app in the way that XCUI does, because in that framework the tests are running in a separate process space from your application.
so every test will start from the same point, regardless of the status in which the previous test finished. This is that Earlgrey lacks, and make all test fails after the first failure.
There's no difference between KIF and EarlGrey in this regard.
From QA pov, each test should be as much scoped as posible. Browsing through views before/after the main actions will increase the risk of failures that are no caused by the target of the test, and then, not-desirable false positives.
I agree. However, as long as we want tests to run inside the apps themselves, so we can also verify internal state, there's really no alternative to this.
Xcode UI tests would supposedly allow relaunching the app between tests, but run in a separate process. Introspection and verifying internal state (like @javiergonzper said he wants to do with row counts of table views if I understood correctly) is then no longer possible.
In the event that one test fails and other, subsequent tests fail because of that, we can see in the logs which test failed first and for what reason. We can then work to fix the issue, so the test no longer fails. If fixing the test takes longer, we can temporarily rename it to start with something other than "test" so Xcode no longer executes it.
i think that tests are developed to avoid such things. One test is intended to cover one scenario. When the test fail, you know that something is broken in the scenario and then, research about it to get the bug or whatever.
It's certainly desirable, but not available with frameworks like EarlGrey or KIF.
As I wrote above, Xcode UI tests are the only ones offering this, but will limit the tests purely to UI elements.
Discussing about
EarlGrey
orKIF
(or other one) is the best, does not have place. We should choose the best for our necessities.
Yeah. That's my point, too.
I will not deep dive in the technical stuff, so i trust in all of you, but what i wrote here #58 (comment) needs to be studied because is one of the first premises to get good and optimal tests.
There's no difference between KIF and EarlGrey in this regard.
I am trying to add more test for the login view but I am having problems due the coupled code between the BookmarkViewController
and the OwncloudSDK
.
I see that the BookmarkViewController
create an object OCBookmark
that is passed to the OwncloudSDK
when is created the object connection
of OCConnection
.
During the execution of connection.prepareForSetup
the initial object OCBookmark
change its properties inside the SDK and then that it is reflected on the BookmarkViewController
.
Things like that make the project difficult to understand and maintain and of course extremely difficult to test.
Is it possible to avoid those kind of practices?
CC: @michaelstingl @jesmrec @felix-schwarz
From a QA pov, the most desirable property is the module isolation, so we can test app and SDK independent each other by mocking, making tests more accurate, useful and maintainable. Please, take it in account in development stage, so mocks will be developed easily and coverage will increase faster.
Any solution for the problem @javiergonzper mentioned?
After add several UI Tests using Swizzling
to mock the responses of SDK methods I found a big problem:
The SDK
do not response always on the method called the result of a query sent the ownCloud Server. Everything related with the list of files is receive in a Delegate
OCCore.createFolder
ownCloud Server
ownCloud Server
OCQueryDelegate
method with the result of the Query
(error, new items...)The problem using Swizzling
is that we mock the first call on 1. Step
to OCCore.createFolder
but there we do not receive the response. The response is received on the OCQueryDelegate
on 6. step
. So mock OCCore.createFolder
do not help.
Remember that when we make Swizzling
to a method we change the internal implementation. All the things that the original method make are not executed.
I talked with @felix-schwarz about this issue and he propose mock the http requests using OCHostSimulator
. It is something very similar to Nocilla library. The idea it is simulate the response of a server when a request it is made to a concrete resource (stubbing). For example if we make a GET to URL http://demo.owncloud.com/ with HEADER XXX... we stub the response returning whatever we want (404, 200 with an XML in the body...).
The problem of make Stubbing using OCHostSimulator
or Nocilla
:
testCreationFolder
could fail because the the SDK fails inserting the user name.cc: @jesmrec @pablocarmu @felix-schwarz @michaelstingl @mneuwert
This is really a point i did not want to reach.
If we go to the OCHostSimulator
or Nocilla
approach, we are transforming UI tests (or tests along the app) to something that will not improve the quality as expected, only covers user scenarios and let you know if something in the middle is wrong, but not the specific error . Every test must have an specific scope, and must be isolated from other dependencies, this is the basis of testing. If i want to test the “Move file": i must create the bookmark, browse, upload the file i want to move, create the target folder, and then execute the action… this is too much noise that will make the tests spend triple or more time to run, and increasing exponentially the odds of wrong positives. Of course, the scenarios can be simplifed (no browse, using skeleton files...). but it would reduce the coverage.
Nocilla
,OCHostSimulator
, and other similar tools are intended to fake dependencies when the test object is linked directly to the network, and this is not the case of ios-app
I really hope that a better solution can be found here.
I do not share these conclusions or fears. Here's why:
OCHostSimulator
: to simulate server responses and the measure/check the reaction along the whole SDK+app stack.OCMocking
: to replace specific methods in the SDK with mocked implementations.Going through the example:
OCCore
as MockCore
OCQuery
as MockQuery
, so that an array of items (mockItems
?) on an instance:
queryHasChangesAvailable()
methodmockItems
when the MockQuery
's requestChangeSet(withFlags:completion:)
method is called in the OCQueryChangeSet
returned via the completion
-block of that MockQuery
methodClientQueryViewController
with instances of MockCore
and MockQuery
From this point on, you can replace all calls to the Core and Query with your own in the subclasses.
Example
- APP -> call to the SDK
OCCore.createFolder
This goes to your own implementation in MockCore.createFolder
. Suggestions on what it can do below.
- APP -> do not receive the response immediately inside the method call (sync or async with a completion)
- SDK -> send the http requests to the
ownCloud Server
- SDK -> receive the response of the
ownCloud Server
- SDK -> process the response updating information on Database
These steps are SDK-internal and - following Jesus' requirement to have isolated tests and best practices - can be skipped.
- SDK -> Call to APP
OCQueryDelegate
method with the result of theQuery
(error, new items...)
MockCore.createFolder
calls the completionHandler
with the desired result and modifies the items in the MockQuery
, so it triggers all the callbacks and provides the modified items (see preparations above for how to implement this in a few lines of code).
cc: @jesmrec @pablocarmu @javiergonzper @michaelstingl @mneuwert
Let me add, too, that all actions - including create folder - are encapsulated in extensions by now, and therefore usable on their own. That's makes even more focused tests possible.
cc: @jesmrec @pablocarmu @javiergonzper @michaelstingl @mneuwert
Right now the only way to use the MockCore or the MockQuery is checking if we are on Testing Mode
to init a OCore or MockCore.
For example:
if IS_TESTING {
core = MockCore.shared.requestCore(for: bookmark, completionHandler: { (_, error) in
if error == nil {
self.coreReady()
}
openProgress.localizedDescription = "Connected.".localized
openProgress.completedUnitCount = 1
openProgress.totalUnitCount = 1
self.progressSummarizer?.stopTracking(progress: openProgress)
})
} else {
core = OCCoreManager.shared.requestCore(for: bookmark, completionHandler: { (_, error) in
if error == nil {
self.coreReady()
}
openProgress.localizedDescription = "Connected.".localized
openProgress.completedUnitCount = 1
openProgress.totalUnitCount = 1
self.progressSummarizer?.stopTracking(progress: openProgress)
})
}
Is it right for you? I do not like it because it is very ugly add if TESTING
on the ViewControllers inside the app.
To do not add if TESTING
in several places I spoke with @pablocarmu and the only possibility that we see is modify the architecture of the app to create the Views inyecting them the necessary objects depending if we are in mock mode or not.
What do you think?
If you have a better idea, please create full example of a real test for us in order to follow it.
Right now as talk on one of our last call I am trying to Subclass the OCCore and OCQuery to create a simple test of creation of a folder.
Using a subclass of OCCore
and OCQuery
I can show a ClientQueryViewController
. That is good because I can update the UI of the ClientQueryViewController
calling to the delgate:
query?.delegate.queryHasChangesAvailable(query!)
Thanks to that I implemented a test of a creation of the folder 👍
Now the problem that I have is that if I just show ClientQueryViewController
I lost the OCCoreDelegate
that it is on the ClientRootViewController
so I can not manage the error.
Maybe I can show the full UI of the file list and then using Swizzling in some point call to the OCCoreDelegate
Subclassed:
MockOCQuery
MockOCCore
MockClientRootViewController
With those change now we can show the file list with the ClientRootViewController (tabs and navigation bar)
I added 6 tests about "Create folder"
This issue is intended to be an epic to discuss about automatic testing.
Earlgrey is the chosen tool to develop UI tests. The branch earlgrey-login contains a first draft of a couple of tests.
We can face two different targets using Earlgrey:
UI testing: check how the app reacts to the SDK's responses. Abstracting dependencies from SDK and mocking responses make posible to isolate the app. This kind of testing complements the unit test in the SDK to have a greater coverage through the full stack of the app.
E2E testing: the scope is the whole app, by using the SDK itself. In this case the whole stack is tested, but is more difficult to abstract tests, and positive results will not give the accurate point of the issue/bug. By using a real server, network/core performance issues can happen causing wrong positives and making the tests flaky.
The point is that we need to abstract dependencies in the app for UI testing, by making testable code. Currently, the app calls directly the SDK interfaces to get info. We should be able to use the same calls with mocked objects, in a way similar to the following one:
for each test. Options to achieve it:
All additional and related ideas are welcome.
@felix-schwarz @javiergonzper @pablocarmu @michaelstingl