Behat tests never end

[seyfer]

Hello

I'm running Behat and Mink extension all latest versions. with Symfony 3.4. Also, I'm using the latest Selenium in Docker image in Hub mode with 2 Chrome nodes. Running in parallel per feature. DB refreshed on every feature run - so tests are isolated per process and per feature. It works well on different suites. But when I run it on a main and biggest suite - it stuck and never ends.

2   51/54   ✔   25 s 249 ms features/backend/3_email_report.feature
1   52/54   ✔   15 s 448 ms features/backend/12_account_select.feature

...

Nevertheless when I run the same suite in normal mode with Behat or in 1 process with fastest - it works, all green.

Trying to think about the issue. The test, which stucks uses iframe on the page. Also, all our scenarios are marked with @javascript tag, so it fails on js error. Could it be the issue with fastest that js error is not recognized? Could it be the issue with use of iframe? Could it be issue with Selenium and hub mode? Any ideas on how to debug?

Part of my behat config

extensions:
    Liuggio\Fastest\Behat\ListFeaturesExtension\Extension: ~
    Behat\Symfony2Extension:
        kernel:
            env: accept
            debug: false
    Behat\MinkExtension:
        base_url: http://gogogo.loc:81/
        selenium2:
            wd_host: http://selenium:4444/wd/hub
            capabilities:
                version: ''
                chrome:
                    switches:
                        - "--disable-gpu"
#                        can be faster, but without VNC
#                        - "--headless"
#                        - "--no-sandbox"
                browser: chrome
                browserName: chrome
        javascript_session: selenium2
        browser_name: chrome
  gherkin:
    cache: ~

[seyfer]

Issue solved. We have some js generated and cached in some directory. Behat running under fastest couldn't access the directory. Running behat with sudo solved the issue.

bin/fastest "sudo bin/behat --profile=default {}"

[seyfer]

Actually, still, that is not clear why fastest wasn't failing on this error?

 And I switch to the iframe "widget-preview"                                                                    # FeatureContext::iSwitchToIframe()
    And I wait for element "iframe[name=da-review-widget]" to appear                                               # FeatureContext::iWaitForElementToAppear()
      Element matching css "iframe[name=da-review-widget]" not found.

If it is not failing and running forever - it will be hell to debug. Can I set some timeouts maybe somewhere in Behat config?

[DonCallisto]

Maybe is JavaScript (ajax) issue?

[DonCallisto]

Btw you can still set a timeout in the waitFor function

[seyfer]

@DonCallisto we are using standard spin function approach as proposed in docs http://docs.behat.org/en/v2.5/cookbook/using_spin_functions.html

I just run the suite with sudo, it still doesn't finish 2 last test. They could be any tests, as the order is random.

1   51/54   ✔   23 s 365 ms features/backend/some/1_builder.feature
1   52/54   ✔   50 s 805 ms features/backend/some/1_widget.feature

53 and 54 are running forever. no status returned.

Another issue: 1 feature from 54 failed at the beginning, but it continued running the rest. There should be a way to stop execution on the first fail.

[DonCallisto]

I'm still not sure that's a fastest issue. With this little code, no real examples, no way to test it, I can't help you at all.

[seyfer]

@DonCallisto well, I'm using all standard configs, nothing special. All configs as in fastest docs. I use provided helper for setting ENV and set ENV in GET and Cookie in FeatureContext in Behat, all works. I use provided helper for changing DB name. Then I have bash script to create DBs with expected names and load dump in them. Config for Selenium and Hub with 2 nodes is standard, as in docs https://github.com/SeleniumHQ/docker-selenium#version-2 I see all works in the console

I run tests with the command find features/backend/ -name *.feature | sort -V -r | bin/fastest "bin/behat --profile=default {}" -p ${PROCESSES_NUMBER} -o --rerun-failed -vv where PROCESSES_NUMBER = 2, for example. I run 2 VNC sessions for both browsers and I see all works in parallel.

But I have 2 issues now.

1. When Behat run with fastest and faces Javascript error - it just stuck, never continues, never fail. This is not happening with normal Behat run.
2. If I run Behat as sudo in order to fix some permissions problems (just testing) then env ENV_TEST_CHANNEL_READABLE is not set and nothing pass as one DB used.

Have no idea why with normal Behat run all tests are green. And paralleled - some fail with JS errors.

Can it be that if 2 processes read the same JS file at the same time - the file is locked for another process? For example, we have in Symfony routing.js to let JS use back-end routes. It loads on all pages. And I see errors like routing.js not found, but that's not true. :)

[DonCallisto]

Again

With this little code, no real examples, no way to test it, I can't help you at all.

I use fastest in my projects and we have javascript scenarios too, so without code - or better, without a reproducible environment - we're not be able to help you more than this.

[DonCallisto]

@seyfer did you resolved this issue?

[seyfer]

@seyfer no I decided to not use it, unfortunately.