Closed maniax89 closed 6 years ago
I had quite a few problems getting the force:lightning:test:run process to work in a docker container under gitlab ci. Just adding some notes to help others.
Under the hood the sfdx client uses the npm package selenium-standalone. (Version 5.7.1 for sfdx 6.0.16.) The --configfile file on the sfdx command line is passed to the selenium-standalone module as the options. So see this module for what you can define in the config file. additionally the following sections of the config file are used :
webdriverio is also passed to the npm webdriverio module as the options to the remote command.
outputDivId is used to wait for the test results. It defines what the test harness is expecting to see on the html page at the end of the testing.
You need to make sure java is avaliable in the path, as well as google-chrome in the docker container. The selenium-standalone npm package automatically downloads selenium and the chromedriver files, you don't need those.
I had a lot of problem trying to debug issues where things seemed to work but nothing happened. The problem was that I couldn't see what the sfdx client was doing. To help i added a shell script in the path before the real java command. This ran java and added some logging to chromedriver as well as re-directing the output of the java command to a file (so i could see it). There doesn't seem to be any other way to grab this output of the command executed by the sfdx client, the stderr and stdout of the subprocess are not captured by the logger.
For example
/usr/local/bin/java:
echo RUNNING java -Djava.util.logging.config.file=config/logging.properties -Dwebdriver.chrome.verboseLogging=verbose -Dwebdriver.chrome.logfile=/tmp/cd.log $@
/usr/bin/java -Djava.util.logging.config.file=config/logging.properties -Dwebdriver.chrome.verboseLogging=verbose -Dwebdriver.chrome.logfile=/tmp/cd.log $@ > /tmp/java.log 2>&1
I got force:lightning:test:run working with headless chrome using the following config file:-
{
"drivers": {
"chrome": {
"version": 2.33,
"arch": "x64",
"baseURL": "https://chromedriver.storage.googleapis.com"
}
},
"requestOpts": {
"timeout": 100000
},
"webdriverio": {
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [
"--headless",
"--disable-gpu",
"--no-sandbox"
]
}
}
}
}
Some notes:
The driver section was necessary as the default driver version 2.23 didn't support chrome headless.
Be careful with the chrome options. I added --snapshot but this caused the broswer load to fail as the option was not compatible with the google --remote-debugging-port flag (which is added automatically by selenium). Some other combination of options failed to start chrome.
In my case the tests that i implemented took a long time, if there is a timeout with testing then you don't get a timeout message. It just fails. So if things appear to be working but you don't get the expected output then it might be a timeout issue.
I had to add the --no-sandbox option above, this isn't a good idea, however i was having a problem with chrome running on a docker debian image. Chrome would fail to run and I would get the following error:-
Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
There seems to be a relevant discussion here https://github.com/jessfraz/dockerfiles/issues/65 . Although i can't quite follow the discussion it seems there are some permissions that are required on the docker and/or the kernel images.
After using the setup you described @innovate42 (which is extremely helpful!) - I still seem to get the least helpful error message:
Invoking Lightning tests using ghenkins-ci... ERROR: Cannot read property 'endsWith' of undefined.
I'm pretty sure I will have to figure out 2 things
google-chrome
onto my docker image (seems like it isn't part of the apt-get
repositories)It would be a lot more helpful if the CLI tool itself helped users debug a bit better
Hi @maniax89 I've found that the endsWith error seems to mean that the command that sfdx-clie was trying to run has failed. The best bet would be to try and see the output - and the only way i've found to do that is to create a wrapper around java as mentioned above.
I've added the Dockerfile that we are using for the test builds here https://github.com/innovate42/dockerfiles/tree/master/sfdx-testenv
endsWith error is a being thrown by sfdx cli while trying to log the actually error see here. Cli team knows about it and its being tracked under bug W-4442231.
When people have encountered it in context of lightning:test:run. It normally has been resolved by setting DISPLAY/Xvfb.
I was able to successfully get things running with https://github.com/forcedotcom/LightningTestingService/issues/46#issuecomment-347928851 after installing google-chrome-stable
It would be good to have this issue archived/turned into documentation so that others may benefit (as it was extremely helpful)
@alderete-sfdc FYI for doc
Finally got everything working; hope this helps others that come behind me. While trying to get aura component testing working with Gitlab CI/CD using a Ubuntu 14.04 Docker image, I had to do the following three key things to get it working:
npm install selenium-standalone@latest -g
and then selenium-standalone install
selenium/standalone-chrome:latest
to services
section in gitlab-ci.yml
config file.Test Command: sfdx force:lightning:test:run -a auraTests.app --configfile config/lts-config.json
lts-config.json
{
"requestOpts": {
"timeout": 100000
},
"seleniumArgs": [
"-host", "selenium__standalone-chrome",
"-port", "4444"
],
"webdriverio": {
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [
"--headless",
"--disable-gpu",
"--no-sandbox"
]
}
},
"host": "selenium__standalone-chrome",
"port": 4444
}
}
gitlab-ci.yml
image: billable/ci-sfdx:latest
services:
- selenium/standalone-chrome:latest
cache:
key: "${CI_COMMIT_REF_SLUG}"
paths:
- node_modules/
variables:
SELENIUM_REMOTE_URL: "http://selenium__standalone-chrome:4444/wd/hub/"
CI_TEMP_DIR: /tmp
SCRATCH_ORG_DEF: config/ci-scratch-def.json
FORCE_APP: force-app/main/default
stages:
- build
- test
- analysis
- packaging
# Environment Setup
before_script:
- sfdx --version
- sfdx plugins --core
- mkdir ${CI_TEMP_DIR}/sfdx-keys
- openssl enc -d -aes-256-cbc -in ./build/assets/server.key.enc -out ${CI_TEMP_DIR}/sfdx-keys/server.key -pass pass:${DEVHUB_SERVER_PASS}
- sfdx force:auth:jwt:grant --clientid ${DEVHUB_CONSUMER_KEY} --jwtkeyfile ${CI_TEMP_DIR}/sfdx-keys/server.key --username ${DEVHUB_USERNAME} --setdefaultdevhubusername -a DevHub
# Build Application
build:
stage: build
script:
- npm install
- npm run build
artifacts:
paths:
- ${FORCE_APP}/aura/baseStyle/baseStyle.css
- ${FORCE_APP}/labels/*.xml
only:
- develop
# Test Automation
test:
stage: test
script:
# Testing Environment Setup
- sfdx force:org:create -s -f ${SCRATCH_ORG_DEF} -a gitlab_build_${CI_JOB_ID}
- sfdx force:lightning:test:install -t jasmine
- sfdx force:source:push -f -u gitlab_build_${CI_JOB_ID}
- sfdx force:user:permset:assign -n Billable_Admin
- mkdir -p "./build/test-results"
# Code Quality
# UI Testing
- source ./build/enable-lts-debug.sh
- npm run test:aura:remote
#- npm run test:lwc
# Apex Unit Testing
- sfdx force:apex:test:run -d ./build/test-results -u gitlab_build_${CI_JOB_ID} -l RunLocalTests -r human -w 10 -c
- node scripts/coverage.js
#Testing Environment Teardown
- sfdx force:org:delete -u gitlab_build_${CI_JOB_ID} -p
coverage: '/^Total\sCoverage:\s+(\d+.\d+\%)$/'
artifacts:
reports:
junit: ./build/test-results/test-*.xml
only:
- develop
for anyone coming back to this comment by @ghost - make sure to use a string (not a decimal) for the version
value under the driver selected - i.e.
{
"drivers": {
"chrome": {
- "version": 2.33,
+ "version": "2.33",
"arch": "x64",
"baseURL": "https://chromedriver.storage.googleapis.com"
}
},
"requestOpts": {
"timeout": 100000
},
"webdriverio": {
"desiredCapabilities": {
"browserName": "chrome",
"chromeOptions": {
"args": [
"--headless",
"--disable-gpu",
"--no-sandbox"
]
}
}
}
}
There doesn't seem to be any way to figure out how to configure the WebDriver settings.
When I run look at the documentation for
sfdx force:lightning:test:run
(https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/cli_reference_force_lightning.htm#cli_reference_test_run) - there is a--configfile
option that says the following:Where is this documentation? I am attempting to run this on a CI tool (like Jenkins) for automated testing and seeing the following error message:
ERROR: not found: java.
I assume it is because I need to install a WebDriver, but there doesn't seem to be any documentation around this. Could you please point me in the right direction?