Failing all tests on Github Actions pipeline

[vitorcamachoo]

Hi, I'm trying to integrate this awesome tool in our company application pipeline. Locally I get all the snapshots Passing, but when it arrives into Github actions, somehow, all the snapshots failed. This is my Loki configuration:

module.exports = {
  diffingEngine: 'looks-same',
  chromeTolerance: 20,
  chromeLoadTimeout: 120000,
  configurations: {
    'chrome.desktop': {
      target: 'chrome.app',
      width: 1920,
      height: 1080,
      deviceScaleFactor: 1,
      mobile: false,
    },
  },
}

And the command that is being executed is:

nx build-storybook backoffice && loki --reactUri file:./dist/storybook/backoffice --chromeFlags='--headless --disable-gpu --hide-scrollbars --no-sandbox'

Am I doing anything wrong ?

[oblador]

Hi, the recommendation is to run the loki tests in docker which will ensure the local and CI environment are identical. Looking at your configuration, the chromeTolerance: 20 sticks out as looking very high.

[vitorcamachoo]

Hi @oblador

The tolerance was a test value to be honest. Don't know if it's 0 to 1 or 0 to 100.

Is loki prepared to be used with puppeteer to avoid using docker ?

If not, where can I get a good example of using loki with docker?

Thanks

[oblador]

Docker is the only way to make sure these tests are deterministic and was kind of the reason I wrote loki to begin with. More libraries have followed, but puppeteer will not help, it’s just a high level abstraction of chrome. Please check out the example in this repo, and the options are documented on the website so you don’t have to guess their values.

[vitorcamachoo]

Hi again, thank for you feedback.

I tried to run on chrome.docker and I'm having the following error:

loki test v0.28.1
 FAIL  chrome.docker
       Fetching stories
       connect ECONNREFUSED 127.0.0.1:58436

[oblador]

Hmm, you're running M1 which I know have had some issues with docker. This error basically says that chrome failed to launch/crashed, this recent article states:

If you are trying to run Intel-based containers on an M1 machine, they can sometimes crash. We are using a piece of software called qemu to emulate Intel chips on M1 but it occasionally fails to run the container. Where possible we recommend sticking to arm64 containers on M1 machines; they will also be faster.

Maybe worth it to try building your own Docker image. I don't have an M1 myself yet, so it's kinda hard to debug. You could try the following command and see if it yields any actionable logs:

docker run --rm -P --security-opt=seccomp=$CWD/node_modules/@loki/target-chrome-docker/src/docker-seccomp.json --shm-size=1g yukinying/chrome-headless-browser:90.0.4408.0 --disable-datasaver-prompt --no-first-run --disable-extensions --remote-debugging-address=0.0.0.0 --remote-debugging-port=61867 --headless --disable-gpu --hide-scrollbar

[vitorcamachoo]

It appears to be related with M1. Running your command I got the following:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
clone: Invalid argument
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

[vitorcamachoo]

I can run the zenika/alpine-chrome:85-with-chromedriver image with success, but I'm getting the following error:

 FAIL  chrome.docker
       Fetching stories
       socket hang up
Some visual tests failed to run

[oblador]

Does zenika/alpine-chrome work in CI?

[vitorcamachoo]

Does zenika/alpine-chrome work in CI?

I'll try and give you some feedback

[vitorcamachoo]

I ran on the CI using chrome.docker option and it runs successfully but giving all the images with differences (as expected). So the thing is, how can I get this work using M1 standardising the options to use chrome.docker ?

[vitorcamachoo]

I tried this on a Windows 10 machine, and I get the same behaviour as M1 Silicon, it gives me timeout on connection to docker

[techeverri]

Could you please provide an example project that consistently shows the error message you mentioned? @vitorcamachoo
I'll run it on my Windows 10 laptop to investigate what causes the error.

[vitorcamachoo]

I finally get it work on Windows using the examples present on Loki repository. It's a M1 Silicon problem. Any suggestion how to get it working between M1 and the other operating systems on my team?

[mcMickJuice]

Following a few of these issues but still struggling to get loki to run in CI/CD. Do any of you have a repo that has a docker file that references an image that will run chrome-docker AND node?

[techeverri]

Do any of you have a repo that has a docker file that references an image that will run chrome-docker AND node?

I'm not sure if this is what you're looking for, but Loki itself runs test in CI (Node.js and Docker) using GitHub Actions (previously using Travis) Example build and configuration used to run Loki in CI 👇

GitHub Actions

• Job 🔗 https://github.com/oblador/loki/runs/1886671748
• Config 🔗 https://github.com/oblador/loki/actions/runs/560801865/workflow

Travis

• Job 🔗 https://cdn.travis-ci.org/github/oblador/loki/builds/681017661
• Config 🔗 https://github.com/oblador/loki/blob/caa4742573e8f98cb79d08428809eddff1e4cbdb/.travis.yml

[mcMickJuice]

@vitorcamachoo did you get CI/CD to work with zenika/alpine-chrome?

When using this image (well, zenika/alpine-chrome:89-with-node specifically), I receive a cryptic cannot call toString of undefined. Looking in the code, this is thrown in the chrome-launcher package as theres no pid on the spawned chrome instance.

I think this is related to how I've setup my docker container so any reference on how you got this to work would be appreciated.

The examples above don't really help me as I:

• need this to run locally in a container thats not chrome.docker (cannot get this to work so I'm using chrome.app)
• we don't use Github Actions or Travis in CI

[techeverri]

I'm closing the issue since the original post is about running Loki using GitHub Action and a working example was provided here https://github.com/oblador/loki/issues/309#issuecomment-823701684

As for the issue mentioned above:

we don't use Github Actions or Travis in CI

It's hard to debug/investigate and fix an issue without a way/playground to consistently reproduce it.

need this to run locally in a container thats not chrome.docker (cannot get this to work so I'm using chrome.app)

If the issue persist, feel free to create a new issue, but please provide and example repo including CI

[Demianeen]

Hello everyone! Have the same problem as @vitorcamachoo that test results on my mac are different from GitHub actions environment, but I already use docker. My loki configuration:

{
  "loki": {
    "configurations": {
      "chrome.laptop": {
        "target": "chrome.docker",
        "width": 1366,
        "height": 768,
        "deviceScaleFactor": 1,
        "mobile": false
      },
      "chrome.iphone7": {
        "target": "chrome.docker",
        "preset": "iPhone 7"
      }
    }
  }
}

The problem that @vitorcamachoo had with ERRORREFUSED with docker I solved with changing loki version to 0.30.0. But tests are still failing, even with docker.

My ci Loki command is build-storybook -c ./config/storybook && npx loki --requireReference --reactUri file:./storybook-static. The problem is in my public repo here (branch login_by_username). Has anyone encountered a similar problem?

[GorillaDevq]

@Demianeen Hello, Dear Friend, Do you solve your problem?

[Demianeen]

@GorillaDevq, nope. I just added threshold for failing test😅