search

wix / Detox

Gray box end-to-end testing and automation framework for mobile apps
https://wix.github.io/Detox/
MIT License
11.15k stars 1.92k forks source link

Headless hardware keyboard default not working #1393

Closed wilau2 closed 5 years ago

wilau2 commented 5 years ago

Describe the bug When running in headless mode with parameter: ConnectHardwareKeyboard boolean NO It behaves like if the connect hardware keyboard was at true.

To Reproduce

Provide the steps necessary to reproduce the issue. If you are seeing a regression, try to provide the last known version where the issue did not reproduce.

  1. Create a test that will fail if connect hardware keyboard was at true. (Usually implies typing in an input)
  2. Run in headless should fail.
  3. Open Simulator.app and should pass.

Expected behavior Headless should behave the same when simulator.app is open

Environment (please complete the following information):

Device and Verbose Detox Logs

visual.release --artifacts-location packages/detox/visual/tmp/ --loglevel verbose
detox[18247] INFO:  [test.js] configuration="ios.visual.release" loglevel="verbose" artifactsLocation="packages/detox/visual/tmp/" recordLogs="none" takeScreenshots="manual" recordVideos="none" recordPerformance="none" node_modules/.bin/jest --config=packages/detox/config.json --maxWorkers=1 '--testNamePattern=^((?!:android:).)*$' packages/detox/visual
detox[18248] INFO:  [DetoxServer.js] server listening on localhost:50132...
detox[18248] DEBUG: [AsyncWebSocket.js/WEBSOCKET_OPEN] opened web socket to: ws://localhost:50132
detox[18248] DEBUG: [DetoxServer.js/LOGIN] role=tester, sessionId=5de147f8-df87-0e58-c7b3-e24446a2aebe
detox[18248] DEBUG: [DetoxServer.js/LOGIN_SUCCESS] role=tester, sessionId=5de147f8-df87-0e58-c7b3-e24446a2aebe
detox[18248] DEBUG: [exec.js/EXEC_CMD, #0] /usr/bin/xcrun simctl list -j
detox[18248] DEBUG: [exec.js/EXEC_CMD, #1] applesimutils --list --byType "iPhone 7" --byOS "12.1"
detox[18248] DEBUG: [exec.js/EXEC_TRY, #1] Searching for device matching iPhone 7...
detox[18248] DEBUG: [exec.js/EXEC_CMD, #2] applesimutils --list --byId "B7ECB144-869C-43D1-BAF9-2D23D358D7F4"
detox[18248] DEBUG: [exec.js/EXEC_CMD, #3] xcodebuild -version
detox[18248] DEBUG: [exec.js/EXEC_CMD, #4] /usr/bin/xcrun simctl boot B7ECB144-869C-43D1-BAF9-2D23D358D7F4
detox[18248] DEBUG: [exec.js/EXEC_TRY, #4] Booting device B7ECB144-869C-43D1-BAF9-2D23D358D7F4
detox[18248] DEBUG: [exec.js/EXEC_CMD, #5] /usr/bin/xcrun simctl bootstatus B7ECB144-869C-43D1-BAF9-2D23D358D7F4
detox[18248] DEBUG: [exec.js/EXEC_CMD, #6] /usr/bin/xcrun simctl uninstall B7ECB144-869C-43D1-BAF9-2D23D358D7F4 org.reactjs.native.example.storybook
detox[18248] DEBUG: [exec.js/EXEC_TRY, #6] Uninstalling org.reactjs.native.example.storybook...
detox[18248] DEBUG: [exec.js/EXEC_SUCCESS, #6] org.reactjs.native.example.storybook uninstalled
detox[18248] DEBUG: [exec.js/EXEC_CMD, #7] /usr/bin/xcrun simctl install B7ECB144-869C-43D1-BAF9-2D23D358D7F4 "/Users/wlauze/github.com/jive/gotoconnect-mobile/packages/storybook/ios/build/Build/Products/Release-iphonesimulator/storybook.app"
detox[18248] DEBUG: [exec.js/EXEC_TRY, #7] Installing /Users/wlauze/github.com/jive/gotoconnect-mobile/packages/storybook/ios/build/Build/Products/Release-iphonesimulator/storybook.app...
detox[18248] DEBUG: [exec.js/EXEC_SUCCESS, #7] /Users/wlauze/github.com/jive/gotoconnect-mobile/packages/storybook/ios/build/Build/Products/Release-iphonesimulator/storybook.app installed
detox[18248] DEBUG: [exec.js/EXEC_CMD, #8] /usr/bin/xcrun simctl terminate B7ECB144-869C-43D1-BAF9-2D23D358D7F4 org.reactjs.native.example.storybook
detox[18248] DEBUG: [exec.js/EXEC_TRY, #8] Terminating org.reactjs.native.example.storybook...
detox[18248] DEBUG: [exec.js/EXEC_SUCCESS, #8] org.reactjs.native.example.storybook terminated
detox[18248] DEBUG: [exec.js/EXEC_CMD, #9] /bin/cat /dev/null >/Users/wlauze/Library/Developer/CoreSimulator/Devices/B7ECB144-869C-43D1-BAF9-2D23D358D7F4/data/tmp/detox.last_launch_app_log.out 2>/Users/wlauze/Library/Developer/CoreSimulator/Devices/B7ECB144-869C-43D1-BAF9-2D23D358D7F4/data/tmp/detox.last_launch_app_log.err && SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="/Users/wlauze/Library/Detox/ios/9038d6671ae06ed92ef33be8da6ef6e58588757b/Detox.framework/Detox" /usr/bin/xcrun simctl launch --stdout=/tmp/detox.last_launch_app_log.out --stderr=/tmp/detox.last_launch_app_log.err B7ECB144-869C-43D1-BAF9-2D23D358D7F4 org.reactjs.native.example.storybook --args -detoxServer "ws://localhost:50132" -detoxSessionId "5de147f8-df87-0e58-c7b3-e24446a2aebe"
detox[18248] DEBUG: [exec.js/EXEC_TRY, #9] Launching org.reactjs.native.example.storybook...
detox[18248] INFO:  [AppleSimUtils.js] org.reactjs.native.example.storybook launched. The stdout and stderr logs were recreated, you can watch them with:
        tail -F /Users/wlauze/Library/Developer/CoreSimulator/Devices/B7ECB144-869C-43D1-BAF9-2D23D358D7F4/data/tmp/detox.last_launch_app_log.{out,err}
detox[18248] DEBUG: [DetoxServer.js/CANNOT_FORWARD] role=testee not connected, cannot fw action (sessionId=5de147f8-df87-0e58-c7b3-e24446a2aebe)
detox[18248] DEBUG: [DetoxServer.js/LOGIN] role=testee, sessionId=5de147f8-df87-0e58-c7b3-e24446a2aebe
detox[18248] DEBUG: [DetoxServer.js/LOGIN_SUCCESS] role=testee, sessionId=5de147f8-df87-0e58-c7b3-e24446a2aebe
LeoNatan commented 5 years ago

I am not familiar with ConnectHardwareKeyboard, but how are you sending the argument? It doesn't seem like it was sent in the log.

wilau2 commented 5 years ago

This is set via xcode ~/Library/Preferences/com.apple.iphonesimulator.plist image

You can see that before xcode 9, detox was booting the simulator with a flag on. It seems to me that this is not working on xcode 10.

https://github.com/wix/Detox/blob/32be293774383d631d0938300175204d4222dd13/detox/src/devices/ios/AppleSimUtils.js#L245

LeoNatan commented 5 years ago

If you add that much argument yourself, does it work?

vadimshvetsov commented 5 years ago

@wilau2 it looks like every developer need to set manually after clonning a repo. Is there a way to pass this argument with Detox settings?

LeoNatan commented 5 years ago

Does it work if passed manually?

LeoNatan commented 5 years ago

Guys, please confirm that it works when adding the launch argument manually to launchApp. If we have a confirmation, we will add this argument.

wilau2 commented 5 years ago

@LeoNatan I'll try that today, yesterday was crazy my bad.

wilau2 commented 5 years ago

So... I've managed to test so far that running tests with the command < xcode 9 works but the simulator does not start headlessly.

So the _bootDeviceMagically command works for xcodeVersion >= 9

I will do some more investigation.

LeoNatan commented 5 years ago

Detox only supports Xcode 10 and above.

vadimshvetsov commented 5 years ago

@wilau2 What command do you use to build and run app with parameter ConnectHardwareKeyboard?

LeoNatan commented 5 years ago

You pass it in launchApp.

wilau2 commented 5 years ago

Tested with: 

await device.launchApp({ launchArgs: { ConnectHardwareKeyboard: false }, ConnectHardwareKeyboard: false });

and

await device.launchApp({ launchArgs: { ConnectHardwareKeyboard: 0 }, ConnectHardwareKeyboard: 0 });

in my init.js

beforeAll(async () => {

})

It's not working in headless mode.

I also ran: https://github.com/evertoncunha/cocoaheads-2018-07/blob/master/fastlane/enable_simulators_keyboards.sh Making sure everything in at 0 by default.

wilau2 commented 5 years ago

I've been reading: https://www.rubydoc.info/gems/simctl/SimCtl/Command/Launch#launch_app-instance_method It looks like there is 2 signature in the launch function and the ConnectHardwareKeyboard arg seems to be supported only by launch_device. launch_app launch_device

I'll try some more stuff.

LeoNatan commented 5 years ago

Why does launch_device do? I am not familiar with this command.

dwilt commented 5 years ago

Any updates on this? We are trying to get detox running in CircleCI and are running into this same issue.

LeoNatan commented 5 years ago

I’ll take a look soon.

dwilt commented 5 years ago

Thanks @LeoNatan. I've worked around it via https://github.com/wix/Detox/issues/39#issuecomment-323515694

LeoNatan commented 5 years ago

Guys, the more I think about it, the more it doesn't make sense to me.

ConnectHardwareKeyboard is only used by the Simulator app. Since we are talking about a headless boot, it shouldn't be read at all and the hardware keyboard should always be disconnected.

dwilt commented 5 years ago

That sounds good to me. Are you suggesting persisting it to being disconnected?

LeoNatan commented 5 years ago

I am suggesting that it is irrelevant in headless simulators.

dwilt commented 5 years ago

Well, it's causing an issue since the soft keyboard is disabled (and doesn't show) because the hardware one is enabled (even though it's a sim). What are you suggesting is the fix?

LeoNatan commented 5 years ago

Why did you decide that it's the hardware keyboard? Have you tried taking a screenshot to make sure that keyboard doesn't indeed show? You can call device.takeScreenshot(name) after tapping on a text field but before trying to type. Please see if the software keyboard is shown or not.

dwilt commented 5 years ago

Yea, let me take a video showing my screen. Will share in a second.

On Mon, Jun 24, 2019 at 8:32 AM Leo Natan notifications@github.com wrote:

Why did you decide that it's the hardware keyboard? Have you tried taking a screenshot to make sure that keyboard doesn't indeed show? You can call device.takeScreenshot(name) https://github.com/wix/Detox/blob/master/docs/APIRef.DeviceObjectAPI.md#devicetakescreenshotname after tapping on a text field but before trying to type. Please see if the keyboard is shown or not.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wix/Detox/issues/1393?email_source=notifications&email_token=AAIEEQ7MRN3H4K6T3C6G323P4DSIVA5CNFSM4HN4X74KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYNKEVI#issuecomment-505061973, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIEEQ4IR5SEX2XYHLLZDDLP4DSIVANCNFSM4HN4X74A .

dwilt commented 5 years ago

hey Leo, just looking into how to take a screenshot now and realizing it's bit involved and since I'm pretty new to CircleCI, it might take me a while.

In the meantime, what I can say is that the errors that are showing up for me in CircleCI seem to indicate that the issue is that the soft keyboard isn't showing. I mean, I can even get it to fail when not in headless mode (actually running the simulator locally). Whenever I start up a brand new simulator, the hardward keyboard is enabled by default - not the soft one. So I would imagine its the same default in headless mode - no matter where it's run.

You would think being able to toggle the hardware keyboard on a simulator shouldn't be possible because as you pointed out - it's a simulator.

LeoNatan commented 5 years ago

But what you are describing is the Simulator.app behavior. That’s different than the simulator runtime. Detox does not launch the app, it only boots the runtime.

To really test, kill your Simulator.app and shutdown all runtimes. Then run a detox test.

You will probably still not reproduce as we know it works. But the tests failing in CI could be many different reasons, so a screenshot would be helpful. You could curl it somewhere or otherwise retrieve it.

dwilt commented 5 years ago

When you say "runtime", you mean the xcodebuild command, right? If I understand correctly, you mean that Detox just runs the xcodebuild command and it's that command that actually fires up the Simulator?

dwilt commented 5 years ago

BTW, I just ran the takeScreenshot command in CircleCI and it would seem it ran:

detox[37851] INFO:  [test.js] configuration="ios.sim.release" cleanup=true artifactsLocation="artifacts/ios.sim.release.2019-06-24 16-00-44Z" recordLogs="none" takeScreenshots="manual" recordVideos="none" recordPerformance="none" reportSpecs=true node_modules/.bin/jest --config=e2e/config.json --maxWorkers=1 '--testNamePattern=^((?!:android:).)*$' "e2e"

I just need to figure out how to access the screenshot..

LeoNatan commented 5 years ago

That's not the screenshots I mean. You should manually trigger a screenshot using device.takeScreenshot(name) in your test, so that it takes a screenshot in the right time.

I am not familiar with Circle. Do they have an artifacts folder where you can put stuff and they appear at the end?

LeoNatan commented 5 years ago

When you say "runtime", you mean the xcodebuild command, right? If I understand correctly, you mean that Detox just runs the xcodebuild command and it's that command that actually fires up the Simulator?

No. I mean simulator runtime.

When Detox runs xcrun simctl boot, it's the simulator runtime (OS process and daemons) that is launched, not the Simulator.app, which is used for interaction with the simulator runtime.

xcodebuild is for compilation, unrelated.

dwilt commented 5 years ago

That's not the screenshots I mean. You should manually trigger a screenshot using device.takeScreenshot(name) in your test, so that it takes a screenshot in the right time. Oh, I thought that was showing me the folder the screenshots would be put in. Here's my code:

await element(by.text('Log in')).tap();

await emailInput.replaceText('dwilt4rville@gmail.com');

const passwordInput = await element(by.id('login-password-input'));
await device.takeScreenshot('typing in password');
await passwordInput.typeText('Airapp123!');

await device.disableSynchronization();
await passwordInput.tapReturnKey();
LeoNatan commented 5 years ago

I think I see the problem.

Try changing your code to

const passwordInput = await element(by.id('login-password-input'));
await passwordInput.tap();
await device.takeScreenshot('typing in password');
await passwordInput.typeText('Airapp123!');
vmurillo commented 5 years ago

I am running into this issue in headless mode; type Text action fails with this message

Keyboard did not appear after tapping on element [E]. Are you sure that tapping on this element will bring up the keyboard?

I am launching my app like this beforeAll(async () => { // reinstall app await device.launchApp({ delete: true, launchArgs: { ConnectHardwareKeyboard: false } }) })

Is there any configuration to be passed in launchApp that ensures the keyboard is always visible when typing a text?

dwilt commented 5 years ago

So the tap really is required first? If so, that should be documented better because on the home page of the docs, it suggests you can just do typeText without the tap in front of it to make sure the keyboard pops up.

All this being said, I put together a video showing our scenario and what's going on. Let me know your thoughts.

IMAGE ALT TEXT HERE

dwilt commented 5 years ago

Update: Something I just noticed was that when I kill the simulator that I've been testing in, the issue with the keyboard goes away. But if I re-run the test without restarting the simulator, the test will fail. Here is my test code:

it('you can login from the landing page and land on the gallery', async () => {
  const emailInput = await element(by.id('login-email-input'));

  await expect(emailInput).toBeNotVisible();

  await element(by.text('Log in')).tap();

  await emailInput.replaceText('dwilt4rville@gmail.com');

  const passwordInput = await element(by.id('login-password-input'));
  await passwordInput.typeText('Airapp123!');

  await device.disableSynchronization();
  await passwordInput.tapReturnKey();

  const loader = await element(by.id('loader'));

  await expect(loader).toExist();

  await waitFor(loader)
    .toNotExist()
    .withTimeout(15000);

  await expect(loader).toNotExist();
  await expect(element(by.id('gallery'))).toBeVisible();
  await device.enableSynchronization();
});

Notice that I'm not using a .tap() before using typeText and it still works. And what's more interesting is that when it succeeds the first time after restarting the simulator, the soft keyboard wasn't enabled. I know this because if I tap into a text field manually, the keyboard doesn't come up. But it did come up during the tests. Again, if I try to run the test without restarting the simulator, the test fails.

dwilt commented 5 years ago

I take that back. It seems inconsistent. Here is a screenshot FYI:

image

This is after I tried to use tap() on the password Input and then right before I used typeText:

    await passwordInput.tap();
    await device.takeScreenshot('typing in password');
    await passwordInput.typeText('Airapp123!');
LeoNatan commented 5 years ago

Thanks for the video!

Tap might not be necessary. (Won't help but shouldn't cause issues.)

OK, let's address the things in the video. First, local running: since you have the Simulator.app open, indeed hiding the software keyboard with CMD+K will break the test. Tap will not be able to show the software keyboard. Everything in the video is the expected behavior when Simulator.app is running.

Now, about circle: the error I saw there was about the loader (when it failed). Just want to make sure that you are seeing actual keyboard failures on CI as well. If you do, it would be really odd to have such difference between runs.

LeoNatan commented 5 years ago

Might have a solution that will fix this issue.

dwilt commented 5 years ago

Ok, another update. All of my tests are passing on CircleCI now which is great. Not sure what changed but it's awesome.

@LeoNatan It's important to note that forcing the keyboard to show (CMD+K) in a local sim is required if any of your events are using typeText. What's interesting is that my headless (circle) tests are passing which sounds like where @vmurillo is having issues - so not sure why it's different. If you'd want to jump on a google hangout and chat, let me know cause I'd be more than happy to make some videos/contribute to the faqs to help document this a bit more.

rotemmiz commented 5 years ago

@dwilt, an unrelated question. Were you able to run Android E2E on circle CI?

dwilt commented 5 years ago

Here's all of my code to share so people can see what I have working in Circle:

e2e/firstTest.spec.js

describe('Air App', () => {
  beforeEach(async () => {
    await device.reloadReactNative();
  });

  it('you can login from the landing page and land on the gallery', async () => {
    const loginButton = await element(by.id('landing-login-button'));
    await loginButton.tap();

    const emailInput = await element(by.id('login-email-input'));
    await emailInput.replaceText('dwilt4rville@gmail.com');

    const passwordInput = await element(by.id('login-password-input'));
    await passwordInput.typeText('Airapp123!');

    await device.disableSynchronization();
    await passwordInput.tapReturnKey();

    const loader = await element(by.id('loader'));
    await expect(loader).toExist();
    await waitFor(loader)
      .toNotExist()
      .withTimeout(15000);
    await expect(loader).toNotExist();

    const gallery = await element(by.id('gallery'));
    await expect(gallery).toBeVisible();

    await device.enableSynchronization();
  });

  it('can perform a text search', async () => {
    const searchButton = await element(by.id('clips-search-button'));
    await searchButton.tap();

    const searchInput = await element(by.id('search-input'));
    await searchInput.typeText('test search');

    await device.disableSynchronization();
    await searchInput.tapReturnKey();
    await device.enableSynchronization();

    const appliedFilter = await element(by.id('search-applied-filter-0'));
    await expect(appliedFilter).toExist();

    const exitSearchButton = await element(by.id('exit-search-button'));
    await exitSearchButton.tap();

    const removedAppliedFilter = await element(by.id('search-applied-filter-0'));
    expect(removedAppliedFilter).toNotExist();
  });
});

.circleci/config.yml

# Use the latest 2.1 version of CircleCI pipeline processing engine, see https://circleci.com/docs/2.0/configuration-reference/
# yml parser for sanity check: https://yaml-online-parser.appspot.com/

version: 2.1

aliases:
  - &restore-cache
    keys:
      - v2-test-{{ .Branch }}-{{ checksum "package.json" }}
      - v2-test-{{ .Branch }}

  - &save-node-cache
    key: v2-test-{{ .Branch }}-{{ checksum "package.json" }}
    paths:
      - node_modules

  - &save-ios-cache
    key: v2-test-{{ .Branch }}-{{ checksum "package.json" }}
    paths:
      - node_modules
      - ios/build

  - &install-node-dependencies |
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
    yarn

defaults: &defaults
  working_directory: ~/airapp

jobs:
  detox:
    <<: *defaults
    macos:
      xcode: '10.2.1'

    steps:
      - checkout
      - run:
          name: Load dependencies
          command: |
            brew update
            brew tap wix/brew
            brew install --HEAD applesimutils
            npm install -g detox-cli
      - run: *install-node-dependencies
      - run:
          name: Fetch CocoaPods Specs
          command: |
            curl https://cocoapods-specs.circleci.com/fetch-cocoapods-repo-from-s3.sh | bash -s cf
            cd ios; pod install
      - run: yarn detox:release

  typescript_check:
    docker:
      - image: circleci/node:8.10
    steps:
      - checkout

      - run:
          name: Install dependencies
          command: *install-node-dependencies

      - run:
          name: Confirm project compiles in TypeScript
          command: yarn type-check

workflows:
  version: 2
  everything:
    jobs:
      - typescript_check
      - detox

And a gif showing it completing locally.

image

AGAIN, to get these to pass, I had to make sure to enable the soft keyboard via CMD+K. If you quit the simulator each time, the keyboard will get reset to being disabled. So if you're going to be continuing to write tests locally, enable the keyboard every time you restart the simulator (or don't quit it if you're only making JS changes).

Not sure why yet but the keyboard seems to work on CircleCI without any issues.

dwilt commented 5 years ago

@rotemmiz we haven't tackled Android yet :sweat_smile:

vmurillo commented 5 years ago

@dwilt so it just started working magically in CircleCI? There was no change in detox config in package.json ? I am currently running tests locally a pressing cmd+K to make it work, I would like to have this running on travisCI on headless mode

LeoNatan commented 5 years ago

With https://github.com/wix/Detox/pull/1480 this will be solved, hopefully. I will merge it now and release a version.

LeoNatan commented 5 years ago

Please install Detox 13.0.0 and try.

dwilt commented 5 years ago

@vmurillo yea, not that I can tell. I did not make any changes that were specific to package.json, any detox config or config.yml (for CircleCI) that I'm aware of that would've made this work.

I'm looking back at my failed CircleCI builds and as you see above, it was regarding another issue - not the keyboard. I was having keyboard issues locally and then seeing failed builds on the server and wrongfully assuming they were keyboard related.