Allow for visibility assertion threshold adjustments in Detox's API

ShivamJoker commented 3 years ago

Describe the bug

I have bottom sheet which I am trying to test by swiping it up and down It seems to be working perfect in iOS but in android I am getting

   Error: Test Failed: 'at least 75 percent of the view's area is displayed to the user.' doesn't match the selected view.
Expected: at least 75 percent of the view's area is displayed to the user.
     Got: "ReactScrollView{id=309, visibility=VISIBLE, width=1054, height=1505, has-focus=false, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.view.ViewGroup$LayoutParams@2c64d3b, tag=commandList, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}"

Steps To Reproduce

[x] I have tested this issue on the latest Detox release and it still reproduces

Add bottom sheet in your app give it a testID
Open the bottom sheet

Check if it's visible (it is indeed)

const commandList = element(by.id('commandList'));
await expect(commandList).toBeVisible();

https://user-images.githubusercontent.com/23727670/126892394-80306b6a-d95d-4de0-9b16-b65fa99810d4.mov

Expected behavior

It should be able to swipe/detect the element like iOS

Detox Trace-Logs

detox[36165] DEBUG: [WSS_CONNECTION, #50549] registered a new connection.
detox[36165] TRACE: [WSS_GET_FROM, #50549] {"messageId":0,"type":"login","params":{"role":"app","sessionId":"d19be7df-eee6-8bee-b5ed-9c847f6faedc"}}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"messageId":0,"type":"loginSuccess","params":{"testerConnected":true,"appConnected":true}}
detox[36165] TRACE: [SESSION_JOINED] app joined session d19be7df-eee6-8bee-b5ed-9c847f6faedc
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"type":"appConnected"}
detox[36165] TRACE: [WS_MESSAGE] {"type":"appConnected"}

detox[36165] TRACE: [WS_SEND] {"type":"isReady","params":{},"messageId":-1000}
detox[36165] TRACE: [WSS_GET_FROM, #tester] {"type":"isReady","params":{},"messageId":-1000}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"isReady","params":{},"messageId":-1000}
detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":-1000,"type":"ready","params":{}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":-1000,"type":"ready","params":{}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":-1000,"type":"ready","params":{}}

detox[11908] TRACE: [ARTIFACTS_LIFECYCLE] artifactsManager.onAppReady({
  deviceId: 'emulator-11074',
  bundleId: 'com.sql_playground',
  pid: 11908
})
detox[36165] TRACE: [DETOX_BEFORE_EACH] running test: "Test Basic UI Open and close the search panel"
detox[36165] TRACE: [ARTIFACTS_LIFECYCLE] artifactsManager.onTestStart({
  title: 'Open and close the search panel',
  fullName: 'Test Basic UI Open and close the search panel',
  status: 'running'
})
detox[36165] TRACE: [WS_SEND] {"type":"invoke","params":{"target":{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.uiautomator.UiAutomator"},"method":"uiDevice","args":[]}},"method":"pressBack","args":[]},"messageId":1}
detox[36165] TRACE: [WSS_GET_FROM, #tester] {"type":"invoke","params":{"target":{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.uiautomator.UiAutomator"},"method":"uiDevice","args":[]}},"method":"pressBack","args":[]},"messageId":1}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"invoke","params":{"target":{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.uiautomator.UiAutomator"},"method":"uiDevice","args":[]}},"method":"pressBack","args":[]},"messageId":1}
detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":-1000,"type":"ready","params":{}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":-1000,"type":"ready","params":{}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":-1000,"type":"ready","params":{}}

detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":1,"type":"invokeResult","params":{"result":true}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":1,"type":"invokeResult","params":{"result":true}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":1,"type":"invokeResult","params":{"result":true}}

detox[36165] TRACE: [WS_SEND] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.EspressoDetox"},"method":"perform","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["search-btn"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxViewActions"},"method":"click","args":[]}}]},"messageId":2}
detox[36165] TRACE: [WSS_GET_FROM, #tester] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.EspressoDetox"},"method":"perform","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["search-btn"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxViewActions"},"method":"click","args":[]}}]},"messageId":2}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.EspressoDetox"},"method":"perform","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["search-btn"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxViewActions"},"method":"click","args":[]}}]},"messageId":2}
detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":2,"type":"invokeResult","params":{"result":null}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":2,"type":"invokeResult","params":{"result":null}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":2,"type":"invokeResult","params":{"result":null}}

detox[36165] TRACE: [WS_SEND] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxAssertion"},"method":"assertMatcher","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["commandList"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForSufficientlyVisible","args":[]}}]},"messageId":3}
detox[36165] TRACE: [WSS_GET_FROM, #tester] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxAssertion"},"method":"assertMatcher","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["commandList"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForSufficientlyVisible","args":[]}}]},"messageId":3}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"invoke","params":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxAssertion"},"method":"assertMatcher","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"androidx.test.espresso.Espresso"},"method":"onView","args":[{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForTestId","args":["commandList"]}}]}},{"type":"Invocation","value":{"target":{"type":"Class","value":"com.wix.detox.espresso.DetoxMatcher"},"method":"matcherForSufficientlyVisible","args":[]}}]},"messageId":3}
detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":3,"type":"testFailed","params":{"details":"'at least 75 percent of the view's area is displayed to the user.' doesn't match the selected view.\nExpected: at least 75 percent of the view's area is displayed to the user.\n     Got: \"ReactScrollView{id=309, visibility=VISIBLE, width=1054, height=1505, has-focus=false, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.view.ViewGroup$LayoutParams@6fd41b1, tag=commandList, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}\"\n"}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":3,"type":"testFailed","params":{"details":"'at least 75 percent of the view's area is displayed to the user.' doesn't match the selected view.\nExpected: at least 75 percent of the view's area is displayed to the user.\n     Got: \"ReactScrollView{id=309, visibility=VISIBLE, width=1054, height=1505, has-focus=false, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.view.ViewGroup$LayoutParams@6fd41b1, tag=commandList, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}\"\n"}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":3,"type":"testFailed","params":{"details":"'at least 75 percent of the view's area is displayed to the user.' doesn't match the selected view.\nExpected: at least 75 percent of the view's area is displayed to the user.\n     Got: \"ReactScrollView{id=309, visibility=VISIBLE, width=1054, height=1505, has-focus=false, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.view.ViewGroup$LayoutParams@6fd41b1, tag=commandList, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}\"\n"}}

    1) Open and close the search panel
detox[36165] TRACE: [DETOX_AFTER_EACH] failed test: "Test Basic UI Open and close the search panel"
detox[36165] TRACE: [ARTIFACTS_LIFECYCLE] artifactsManager.onTestDone({
  title: 'Open and close the search panel',
  fullName: 'Test Basic UI Open and close the search panel',
  status: 'failed',
  timedOut: false
})

detox[36165] TRACE: [ARTIFACTS_LIFECYCLE] artifactsManager.onBeforeCleanup()
detox[36165] TRACE: [WS_SEND] {"type":"cleanup","params":{"stopRunner":false},"messageId":-49642}
detox[36165] TRACE: [WSS_GET_FROM, #tester] {"type":"cleanup","params":{"stopRunner":false},"messageId":-49642}
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"cleanup","params":{"stopRunner":false},"messageId":-49642}
detox[36165] TRACE: [WSS_GET_FROM, #app] {"messageId":-49642,"type":"cleanupDone","params":{}}
detox[36165] TRACE: [WSS_SEND_TO, #tester] {"messageId":-49642,"type":"cleanupDone","params":{}}
detox[36165] TRACE: [WS_MESSAGE] {"messageId":-49642,"type":"cleanupDone","params":{}}

detox[36165] TRACE: [SESSION_TORN] tester exited session d19be7df-eee6-8bee-b5ed-9c847f6faedc
detox[36165] TRACE: [WSS_SEND_TO, #app] {"type":"testerDisconnected","messageId":-1}
detox[36165] TRACE: [SPAWN_KILL, #36217] sending SIGINT to: /Users/shivam/Library/Android/sdk/platform-tools/adb -s emulator-11074 shell am instrument -w -r -e detoxServer ws://localhost:50462 -e detoxSessionId d19be7df-eee6-8bee-b5ed-9c847f6faedc -e debug false com.sql_playground.test/androidx.test.runner.AndroidJUnitRunner
detox[36165] DEBUG: [SPAWN_END, #36217] /Users/shivam/Library/Android/sdk/platform-tools/adb -s emulator-11074 shell am instrument -w -r -e detoxServer ws://localhost:50462 -e detoxSessionId d19be7df-eee6-8bee-b5ed-9c847f6faedc -e debug false com.sql_playground.test/androidx.test.runner.AndroidJUnitRunner terminated with SIGINT
detox[36165] DEBUG: [EXEC_CMD, #26] "/Users/shivam/Library/Android/sdk/platform-tools/adb" -s emulator-11074 reverse --remove tcp:50462
detox[36165] DEBUG: [WSS_CLOSE] Detox server has been closed gracefully

Device logs (adb logcat)

Logs - https://gist.github.com/ShivamJoker/cb00dbf8c243f038fdb14d660adbac58

Screenshots

Environment (please complete the following information):

Detox: "^18.19.0"
React Native: "0.64.2"
Node: "15.2.0"
Device: Pixel_API_29_AOSP
OS: macOS 11.4 20F71 x86_64
Test-runner mocha

jonathanmos commented 3 years ago

@ShivamJoker we will investigate this. You can try in the meantime as a workaround to check the visibility of the first item inside the bottomsheet instead of the bottomsheet itself

d4vidi commented 3 years ago

@jonathanmos this brings up very old discussions regarding custom visibility thresholds (i.e. apply such in our various API's). Perhaps we should reconsider. In any case, I wonder why this works flawlessly on iOS (no visibility constraint for swipe actions?) @alon-ha we could use you advice

badsyntax commented 3 years ago

I was receiving the same error but in a different scenario. I don't want to distract from OP's issue but thought this information might be useful for others.

I was doing this in my global jest setup file:

beforeEach(async () => {
  await device.reloadReactNative();
});

And detox was not able to identify android as being "stable" before running the tests and was throwing the "75%" error when checking for visibility of elements. I was only getting this error a low powered VM in CI (Azure DevOps Pipelines), I could not replicate this error locally, and also this error only occurred on my Android tests, no error for iOS tests.

I took some screenshots before running my visibility assertions and this is the state of the Android app:

To fix I removed await device.reloadReactNative(); from the global beforeEach and am now manually restoring state of the app after each test.

d4vidi commented 3 years ago

Thanks @badsyntax. The error is very generic and can stem from a vast amount of erroneous use cases. As far as you can tell, did Detox move forward to start executing your it() code before the app was idle? Is it possible that the app fails to load, on occasions? Please visit your device.log files and check for errors 🙏🏻

alon-ha commented 3 years ago

@jonathanmos this brings up very old discussions regarding custom visibility thresholds (i.e. apply such in our various API's). Perhaps we should reconsider. In any case, I wonder why this works flawlessly on iOS (no visibility constraint for swipe actions?) @alon-ha we could use you advice

There is a visibility constraint (0.75), so same as android. My guess is that in iOS the bottom sheet open fully (DetoxSynch wait until the animation finish), and only then the visibility test is occurring) @ShivamJoker what is exactly the order of the test? Do you open the bottom sheet and only then try to check the visibility? (in such case iOS seems to work as it should). It will be good if you can post the code for the test here.

ShivamJoker commented 3 years ago

@alon-ha I am first clicking on the search button and then it opens the bottom sheet, then I tried to check it's visibility and also tried swiping it up and down.

Edit - also check the video demo it will clearly tell whats happening

The code is this

 it('Open and close the search panel', async () => {
   await tapOnId('search-btn');
   await element(by.id('searchBox')).tap();

   const commandList = element(by.id('commandList'));
   await expect(commandList).toBeVisible();
   await commandList.swipe('up');
   await commandList.swipe('down');
   await commandList.swipe('down');
});

You can check the repo as well - https://github.com/ShivamJoker/SQL-Play/blob/master/e2e/firstTest.spec.js

alon-ha commented 3 years ago

@ShivamJoker thanks for the quick reply. @d4vidi From the code, it supposed to work both on iOS and Android. After clicking the search button, the system should wait until the bottom sheet is fully opened and only then try to check the await expect(commandList).toBeVisible(); From the screenshoot in the emulator, it seems that the moment it failed (i.e the visibility check was already performed), the bottom sheet wasn't fully opened yet.

d4vidi commented 3 years ago

Sorry, I was getting action visibility constraints mixed up with simple visibility assertions.

I checked the video + logs and I think the conclusion is simple: that the view is indeed not 75% visible... Perhaps on the target iOS device the different device dimensions make the command list view visible, just above the threshold.

In any case, one thing I noticed was that RN's debug alerts bar blocks a part of the view. May I suggest - retry this in release mode, where the bar does not appear. If that fails, another direction is to use an emulator with a higher resolution, or - as suggested, assert the visibility of one of the inner command items. Last but not least, undeniably, the best option would be to fix this in the framework, as initially suggested. However, I can't guarantee any timelines ATM.

ShivamJoker commented 3 years ago

@d4vidi the thing is I even I tried changing the bottom sheet height to 90% and got the same error. I'll try once again and share the video.

Are you sure that changing the threshold will solve the problem and it isn't what @alon-ha is saying “it's not waiting for the bottom sheet animation to finish before the visibility check”

d4vidi commented 3 years ago

@ShivamJoker well that's pretty easy to test - add an await sleep() after tapping the button. Let us know. BTW:

const sleep(ms) = new Promise((resolve) => setTimeout(resolve, ms));

🤷🏻

ShivamJoker commented 3 years ago

Okay @d4vidi, it seems to work if I set the bottom sheet height to 90% So I think I have to wait for this feature to be included and for now I will try to check the first element in the list.

Let me know if I can do anything else to contribute further.

asafkorem commented 2 years ago

@ShivamJoker thanks for posting this feature request! We exposed the visibility threshold as an optional parameter (see toBeVisible(pct?: number)). It will be released in the next version of Detox.

ShivamJoker commented 2 years ago

@asafkorem that's great news. Hope it will solve our issues 🎉

ShivamJoker commented 2 years ago

@asafkorem it is working great but can anyone fix the type for toBeVisible ?

typescript: Property 'toBeVisible' does not exist on type 'JestMatchers'.

Update: it seems to have properties if I import it from detox rather than using the global version

asafkorem commented 2 years ago

@ShivamJoker do you still have this error message? Also, what do you mean by "the global version"?

ShivamJoker commented 2 years ago

@asafkorem I meant if I don't require the expect from detox library it still shows the error that the property doesn't exist

If I do this the error goes

const { expect } = require('detox');

The default expect is being taken from jest

wix / Detox