`percy start` blocks test scripts

[johanrd]

Hi.

I'm trying to get four parallell test scripts to post to the same build. According to the docs, it should be a matter of:

• percy start is called (and creates a Percy build)
• Tests run in parallel, on the same machine
• Tests complete
• percy stop is called to finalize the Percy build

From this, I would have expected to be able to run npx percy start – then my test scripts – then npx percy stop. However, when I'm running npx percy start in the mac os terminal, the process is completely blocked after:

$ npx percy start
[percy] created build #146: https://percy.io/c1278ae5/StreamBIM-webapp/builds/7685040
[percy] percy has started.
[percy] percy is ready.
…
$ Not possible to run any command here without stopping percy

I have not found any examples in the documentation of percy start – almost all examples are wrapped in percy exec (which makes test scripts end up finalized in separate builds). Is there a complete example in the documentation with percy start and percy stop/percy finalize --all?

Thanks in advance! johan

[johanrd]

Ok, so I somehow got it to work with the following setup:

$ export PERCY_PARALLEL_NONCE=`date +%s`
$ yarn percy start & yarn test

// package.json

"scripts": {
    "test": "npm-run-all --parallel test:* && yarn percy finalize --all",
    "test:mobilePortrait": "DEVICE=mobilePortrait ember exam --test-port=7357 --path='dist'",
    "test:mobileLandscape": "DEVICE=mobileLandscape ember exam --test-port=7358 --path='dist'",
    "test:tablet": "DEVICE=tablet ember exam --test-port=7359 --path='dist'",
    "test:desktop": "DEVICE=desktop ember exam --test-port=7360 --path='dist'",
  },

They all now process correctly under the same build, however the setup feels quite hackish as the server at port 5338 is not stopped: When rerunning the script I get:

Error: listen EADDRINUSE: address already in use :::5338
{
  code: 'EADDRINUSE',
  errno: -48,
  syscall: 'listen',
  address: '::',
  port: 5338
}

yarn percy stop outputs

[percy] percy is already stopped.

however the server is still up.

For now I can kill it manually by lsof -i TCP:5338 | grep LISTEN and then kill -9 [node], but there must be a better way…

[wwilsman]

Hey @johanrd!

The start command does not automatically run in the background. You'll have to do that by using the -d or --detached flag (percy start -d). Your yarn percy start & yarn test works because the & is actually a command line operator that tells it to run the first command in the background. If you used &&, the logical and operator, then your tests would not run until the start command has exited successfully.

The reason why the stop command might claim it is already stopped is due to how it rudimentarily checks for a PID file that is created only when using the detached flag. Without the detached flag, the only way to stop the start command is by killing it like you are manually doing with kill.

The parallel docs are a little confusing because there are actually a couple of ways to achieve parallel builds with Percy.

1. If there is both PERCY_PARALLEL_NONCE and PERCY_PARALLEL_TOTAL environment variables, it will automatically wait for the total number of builds that have the same nonce to be complete before it considers the entire build finalized.

2. If you set PERCY_PARALLEL_TOTAL=-1 (negative one), that will signal Percy to wait for you to manually trigger finalize --all to finalize all of the parallel builds once all of your parallel tests have finished.

Both of these methods are meant to help when you have Percy itself running in parallel. i.e. if percy exec is called multiple times across different parallel machines. If only your tests are running in parallel from the same process on the same machine, then you do not need to parallelize Percy itself.

1. Within the same process, percy exec -- yarn test should just work. This command is roughly the equivalent of running percy start -d && yarn test && percy stop. If all of your parallel tests happen within yarn test and are finished when that command finishes, then this is what you are looking for.

2. If your tests are running across multiple processes but on the same machine, then this is when you can manually use the start command, run each of your tests, then use the stop command to finalize that build.

To summarize: You should not need to set any PARALLEL options, or use the finalize command, unless you are calling the exec command multiple times (and parallelizing Percy itself). And you should not need to use the start and stop commands unless your tests are running in multiple parallel processes on the same machine.

Judging from your scripts, you don't need to parallelize Percy itself. So those PARALLEL variables and the finalize command are not needed. The npm-run-all package should run all of your parallel tests in it's own process as sub-processes. So you should be able to use yarn percy exec -- yarn test after removing the finalize command from that script.

[johanrd]

Hey @wwilsman!

Thanks a lot for clarifying – I really appreciate the thorough reply:heart:

To sum up – for anyone stumbling by – the following seems to be a performant setup:

// package.json
"scripts": {
    "build-for-test": "ember build --output-path='dist' --environment=test",
    "test": "npm-run-all --parallel test:*",
    "test:mobilePortrait": "DEVICE=mobilePortrait ember exam --test-port=7357 --path='dist'",
    "test:mobileLandscape": "DEVICE=mobileLandscape ember exam --test-port=7358 --path='dist'",
    "test:tablet": "DEVICE=tablet ember exam --test-port=7359 --path='dist'",
    "test:desktop": "DEVICE=desktop ember exam --test-port=7360 --path='dist'",
  }

$ yarn build-for-test
$ yarn percy exec yarn test

Thanks again!