Testing live views with chromedriver fails on GitHub actions

[maltekrupa]

Issue

Hi,

I hope this actually is an issue, because so far the only thing I can say is: "works on my machine".😢

In order to reproduce it, I setup a test repository: https://github.com/maltekrupa/wallaby_test_github_actions

What was done so far in this repository (breaking down the commits):

1. mix phx.new wallaby_test_github_actions --no-ecto
2. Add a simple live view
3. Install and configure wallaby
4. Add a test for said live view
5. Confirm it works locally (it does 🤷)
6. Add GitHub action
7. Confirm it fails on GitHub actions

And now, for whatever reason, the second assert_has fails on GitHub after virtually clicking on the Increment button.

Details

Locally

Elixir: 1.13.4 Erlang: 25.0.2 OS: MacOS 12.4 Webdriver (selenium, chromedriver, geckodriver, etc): chromedriver 103.0.5060.53 Browser (Chrome, Firefox, Safari, etc): Google Chrome 103.0.5060.114 Local or CI: Local

GitHub actions

Elixir: 1.13.4 Erlang: 25.0.2 OS: ubuntu-latest on GitHub actions Webdriver (selenium, chromedriver, geckodriver, etc): chromedriver 103.0.5060.53 Browser (Chrome, Firefox, Safari, etc): Google Chrome 103.0.5060.114 Local or CI: CI

Test Code & HTML

Test

Source

defmodule WallabyTestGithubActionsWeb.IndexLiveTest do
   use ExUnit.Case, async: true
   use Wallaby.Feature

   feature "can increment number", %{session: session} do
     session
     |> visit("/wtga")
     |> assert_has(Query.css("h3", text: "0"))
     |> click(Query.button("Increment"))
     |> assert_has(Query.css("h3", text: "1"))
     |> click(Query.button("Increment"))
     |> click(Query.button("Increment"))
     |> assert_has(Query.css("h3", text: "3"))
   end
 end

HTML

Source

    <div class="container">
      <form action="#" phx-submit="increment" novalidate>
        <%= submit "Increment", phx_disable_with: "Incrementing..." %>
      </form>
    </div>
    <h3><%= @counter %></h3>

Project that reproduces the bug

https://github.com/maltekrupa/wallaby_test_github_actions

Random GIF

[mhanberg]

Absolutely amazing bug report!

I'll take a look soon 👍.

[mhanberg]

Would you mind turning on automatic screenshots and adding this to the GitHub action so that we can see the screenshot?

      - uses: actions/upload-artifact@v2
        if: failure()
        with:
          name: Wallaby Screenshots
          path: ./screenshots/

[maltekrupa]

Absolutely amazing bug report!

It's the least I can do. :)

Would you mind turning on automatic screenshots [..]

Not at all. Here you go: https://github.com/maltekrupa/wallaby_test_github_actions/actions/runs/2686408209

I also added you as a collaborator so you can access the artifacts and (if needed) change whatever needs to be changed in the repo.

[maltekrupa]

Just saw your fork. Even better.

Thank you for investigating!

[mhanberg]

https://github.com/mhanberg/wallaby_test_github_actions/runs/7379446012?check_suite_focus=true

You need to remember to compile your JS assets when running your tests. You can see that this commit fixes the issue.

[maltekrupa]

Argh, thank you! It even says so in the readme. Sorry.

[mhanberg]

No worries, thanks again for the detailed report 👍