Closed cklogs closed 2 years ago
@cklogs, karma-coverage
needs to be configured using the coverageReporter
property, which is missing from your configuration.
Can you please try to change the configuration like the below;
// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html
module.exports = function (config) {
config.set({
basePath: '',
frameworks: ['jasmine', '@angular-devkit/build-angular'],
plugins: [
require('karma-spec-reporter'),
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-coverage'),
require('@angular-devkit/build-angular/plugins/karma')
],
client: {
clearContext: false // leave Jasmine Spec Runner output visible in browser
},
+ coverageReporter: {
+ dir: require('path').join(__dirname, './coverage/karma-coverage'),
+ subdir: '.',
+ reporters: [
+ { type: 'html' },
+ { type: 'text-summary' }
+ ]
+ },
- coverage: {
- dir: require('path').join(__dirname, '../coverage/karma-coverage'),
- reports: ['html', 'lcovonly', 'text-summary']
- },
reporters: ['dots', 'spec', 'coverage'],
specReporter: {
maxLogLines: 5, // limit number of lines logged per test
suppressErrorSummary: true, // do not print error summary
suppressFailed: false, // do not print information about failed tests
suppressPassed: true, // do not print information about passed tests
suppressSkipped: true, // do not print information about skipped tests
showSpecTiming: false, // print the time elapsed for each spec
failFast: false // test would finish with error when a first fail occurs.
},
customLaunchers: {
'CustomChromeHeadless': {
base: 'ChromeHeadless',
flags: [
'--disable-gpu',
'--disable-web-security',
'--no-sandbox',
'--remote-debugging-port=9222'
],
debug: true
}
},
browsers: ['CustomChromeHeadless'],
browserDisconnectTolerance: 2,
browserNoActivityTimeout: 300000, // five minutes
browserDisconnectTimeout: 60000,
captureTimeout: 300000,
hostname: '127.0.0.1',
port: 9876,
colors: true,
logLevel: config.LOG_DEBUG,
autoWatch: false,
singleRun: true,
restartOnFileChange: false
});
};
More information about karma-coverage configuration can be found here: https://github.com/karma-runner/karma-coverage/blob/master/docs/configuration.md
@alan-agius4
I tried exactly as is suggested above and coverage reports are still not being generated. Perhaps there is something more I'm missing?
Without a reproduction it's hard to tell what's happening. You can try to debug https://github.com/karma-runner/karma-coverage/blob/1d34a56ba612ee3970c99c2e2a4822a50870384c/lib/reporter.js#L224 to see what's happening.
I'm still trying to demystify what is going on but I have observed the following: For these particular methods:
The browsers
object on line 185
is defined as:
BrowserCollection {
browsers: [],
emitter: EventEmitter {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
[Symbol(kCapture)]: false
}
}
This means that if (browsers) {
executes.
However, this.onBrowserStart
is never called during the entire execution of the tests.
Once the tests have all ran, this function is called:
The parameter browser
is defined as:
Browser {
id: '35804940',
fullName: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/89.0.4389.90 Safari/537.36',
name: 'Chrome Headless 89.0.4389.90 (Windows 10)',
lastResult: BrowserResult {
startTime: 1617310253880,
total: 0,
success: 6032,
failed: 0,
skipped: 0,
totalTime: 628830,
netTime: 437749,
error: false,
disconnected: false
},
disconnectsCount: 0,
activeSockets: [
Socket {
_events: [Object: null prototype],
_eventsCount: 8,
_maxListeners: undefined,
nsp: [Namespace],
client: [Client],
acks: Map {},
fns: [],
flags: {},
_rooms: Set {},
server: [Server],
adapter: [Adapter],
id: 'eEVk_hCsxP6b3WzbAAAD',
connected: true,
disconnected: false,
handshake: [Object],
[Symbol(kCapture)]: false
}
],
noActivityTimeout: 300000,
singleRun: true,
clientConfig: {
args: [],
useIframe: true,
runInParent: false,
captureConsole: true,
clearContext: true,
originalArgs: []
},
collection: BrowserCollection {
browsers: [ [Circular] ],
emitter: Server {
_events: [Object: null prototype],
_eventsCount: 17,
_maxListeners: undefined,
log: [Logger],
loadErrors: [],
_injector: [Injector],
_boundServer: [Server],
_fileList: [FileList],
[Symbol(kCapture)]: false
}
},
emitter: Server {
_events: [Object: null prototype] {
load_error: [Function],
exit: [Array],
run_complete: [Array],
browser_complete: [Array],
file_list_modified: [Array],
run_start: [Array],
browser_start: [Array],
browser_error: [Array],
browser_log: [Array],
spec_complete: [Array],
spec_failure: [Function],
browsers_change: [Function],
browser_register: [Function],
stop: [Function],
browser_restart_failure: [Function],
browser_complete_with_no_more_retries: [Function],
browser_process_failure: [Function]
},
_eventsCount: 17,
_maxListeners: undefined,
log: Logger {
category: 'karma-server',
context: {},
parseCallStack: [Function: defaultParseCallStack]
},
loadErrors: [],
_injector: Injector {
_providers: [Object: null prototype],
_instances: [Object: null prototype],
get: [Function: get],
invoke: [Function: invoke],
instantiate: [Function: instantiate],
createChild: [Function: createChild]
},
_boundServer: Server {
_events: [Object: null prototype],
_eventsCount: 3,
_maxListeners: undefined,
_connections: 0,
_handle: [TCP],
_usingWorkers: false,
_workers: [],
_unref: false,
allowHalfOpen: false,
pauseOnConnect: false,
_connectionKey: '4:0.0.0.0:9876',
[Symbol(kCapture)]: false,
[Symbol(asyncId)]: 13
},
_fileList: FileList {
_patterns: [Array],
_excludes: [Array],
_emitter: [Circular],
_preprocess: [AsyncFunction: preprocess],
buckets: [Map],
_refreshing: [Promise],
_emitModified: [Function]
},
[Symbol(kCapture)]: false
},
socket: Socket {
_events: [Object: null prototype] {
start: [Function],
info: [Function],
karma_error: [Function],
result: [Function],
complete: [Function],
error: [Function],
register: [Function],
disconnect: [Function]
},
_eventsCount: 8,
_maxListeners: undefined,
nsp: Namespace {
_events: [Object: null prototype],
_eventsCount: 1,
_maxListeners: undefined,
sockets: [Map],
_fns: [],
_rooms: Set {},
_flags: {},
_ids: 0,
server: [Server],
name: '/',
adapter: [Adapter],
[Symbol(kCapture)]: false
},
client: Client {
sockets: Map {},
nsps: Map {},
server: [Server],
conn: [Socket],
encoder: Encoder {},
decoder: [Decoder],
id: '2DlQ3r3gH89vkFHCAAAA',
onclose: [Function: bound onclose],
ondata: [Function: bound ondata],
onerror: [Function: bound onerror],
ondecoded: [Function: bound ondecoded],
connectTimeout: undefined
},
acks: Map {},
fns: [],
flags: {},
_rooms: Set {},
server: Server {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
_nsps: [Map],
parentNsps: Map {},
_path: '/socket.io',
clientPathRegex: /^\/socket\.io\/socket\.io(\.min|\.msgpack\.min)?\.js(\.map)?$/,
_connectTimeout: 45000,
_serveClient: true,
_parser: [Object],
encoder: Encoder {},
_adapter: [class Adapter extends EventEmitter],
sockets: [Namespace],
opts: [Object],
eio: [Server],
httpServer: [Server],
engine: [Server],
[Symbol(kCapture)]: false
},
adapter: Adapter {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
nsp: [Namespace],
rooms: [Map],
sids: [Map],
encoder: Encoder {},
[Symbol(kCapture)]: false
},
id: 'tpLs3SFHdGeiHnz_AAAB',
connected: false,
disconnected: true,
handshake: {
headers: [Object],
time: 'Thu Apr 01 2021 13:50:53 GMT-0700 (Pacific Daylight Time)',
address: '127.0.0.1',
xdomain: false,
secure: false,
issued: 1617310253837,
url: '/socket.io/?EIO=4&transport=polling&t=NYFD0BQ',
query: [Object: null prototype],
auth: {}
},
[Symbol(kCapture)]: false
},
timer: {
setTimeout: [Function: setTimeout],
clearTimeout: [Function: clearTimeout]
},
disconnectDelay: 60000,
log: Logger {
category: 'Chrome Headless 89.0.4389.90 (Windows 10)',
context: {},
parseCallStack: [Function: defaultParseCallStack]
},
noActivityTimeoutId: Timeout {
_idleTimeout: 300000,
_idlePrev: [TimersList],
_idleNext: [TimersList],
_idleStart: 983207,
_onTimeout: [Function],
_timerArgs: undefined,
_repeat: null,
_destroyed: false,
[Symbol(refed)]: true,
[Symbol(asyncId)]: 1889860,
[Symbol(triggerId)]: 1889856
},
pendingDisconnect: Timeout {
_idleTimeout: -1,
_idlePrev: null,
_idleNext: null,
_idleStart: 385875,
_onTimeout: null,
_timerArgs: undefined,
_repeat: null,
_destroyed: true,
[Symbol(refed)]: true,
[Symbol(asyncId)]: 1809518,
[Symbol(triggerId)]: 1809514
},
state: 'CONNECTED'
}
coverageMaps
are defined as:
[Object: null prototype] {}
The result
is a large object containing coverage results for each file. One such example:
'C:\\app\\shared\\helpers\\url-serializer.ts': {
path: 'C:\\app\\shared\\helpers\\url-serializer.ts',
statementMap: [Object],
fnMap: [Object],
branchMap: [Object],
s: [Object],
f: [Object],
b: [Object],
inputSourceMap: [Object],
_coverageSchema: '1a1c01bbd47fc00a2c39e90264f33305004495a9',
hash: 'b12799a1103f348e9c0f91bb2cb5bbf81c79d909'
}
Is there anything more you would like to see?
@alan-agius4 When you have a chance, could you please review my updates? Advance thanks!
I'll leave the triaging to someone from the karma-coverage team.
To the maintainers of this repo:
I have a small suspicion that,
have something to do with coverage failing to be generated. But this is only a hunch.
I cannot share or anonymize our code base, but if there is a large open repository that you would suggest I try and reproduce this issue from, I will happily try.
Aside from that, if you have any insights on the above issue, please let me know here or contact me directly.
Thanks again.
We also have a big project with 3000+ unit tests. And I can see coverage reports generated only on powerful machines. On our ci server and on slow machines it's not generated.
@WebMex Thank you for the additional data point!
Do you plan to continue using karma-coverage, or have you sought out a different library? We have not been able to move onto karma-coverage (from karma-coverage-instanbul-reporter) because of the above issue, but we are open to other coverage generators.
@cklogs I have rolled back to karma-coverage-instanbul-reporter so far and waiting for the fix. Despite the warning message from angular it is still possible to use it.
@alan-agius4
So far I have had little luck getting feedback from any of the maintainers of this library. Given karma-coverage is now the defacto standard coverage library for Angular, it may be necessary to raise this up to the greater Angular team.
Please advise on how to most effectively do this.
Thanks in advance for your help!
@WebMex
Thanks to your tip, I tested karma-coverage on a faster box (Ryzen 7 PRO / 32 GB RAM), and I am getting consistent coverage results there. Still need this fix for our deployment servers to generate coverage, but I'm hoping these multiple data points give the maintainers a lead on the issue.
This would be a hard one to track down without a reproduction. I tried it on a seed project and generated a large number of tests but I didn't manage to replicate.
Unfortunately, I am not too familiar with this plugin to get to the bottom of this without being able to reproduce the issue.
I am seeing the same issue. If I run the test on my mac locally, I'm seeing the lcov.info file always get generated. However, running the test with even a greater --max_old_space_size
size specified and run by Jenkins on a linux box intermittently does not generate the lcov.info file. There are no reports of exceptions or anything else I can see that explain why the report is not generated.
When the coverage isn't generated I see something like:
[2021-06-29T21:37:31.258Z] TOTAL: 6266 SUCCESS
[2021-06-29T21:37:31.258Z]
[2021-06-29T21:37:31.258Z] Finished in 13 mins 7.178 secs / 9 mins 58.633 secs @ 21:37:31 GMT+0000 (Coordinated Universal Time)
[2021-06-29T21:37:31.258Z]
[2021-06-29T21:37:31.258Z] SUMMARY:
[2021-06-29T21:37:31.258Z] ✔ 6266 tests completed
[2021-06-29T21:37:31.258Z] ℹ 28 tests skipped
When I run it locally and the file is generated I get:
TOTAL: 6266 SUCCESS
Finished in 14 mins 0.813 secs / 10 mins 58.549 secs @ 15:34:47 GMT-0600 (Mountain Daylight Time)
SUMMARY:
✔ 6266 tests completed
ℹ 28 tests skipped
29 06 2021 15:35:04.211:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
Oddly, it seems the Chrome was not killed
is usually reported when the lcov.info
file is generated, but if that is not reported, the coverage file was not generated. So it almost seems like the node process is exiting before the karma-coverage plugin is done generating the report.
Could the problem be that the process can exit before the plugin/reporter has completed writing the file?
@alan-agius4 Perhaps this project might be a good place to run a smoke test of this issue: https://github.com/angular/angular-cli-stress-test. I was able to get it roughly running just by dropping the files into a newly spun up Angular 12 project. I could not get coverage reports generating on the codebase. You will likely be able to reproduce the issue so long as your machine is not too powerful.
Please can this be done? We have the same issue. We have about 13000 unit tests. Sometimes the coverage report gets generated, however it mostly fails. We run them on a Linux docker image trion/ng-cli-karma.
It's upsetting to be "punished" for having too many tests.
I have done a previous investigation on this, and I found that it was related to the speed/number of tests. It failed in Istanbul, I believe it was when merging the coverage summaries that the result was just suddenly empty and then no coverage were created
Unfortunately this started happening ~75% of the time for us, but I could not reproduce it locally. Only solution I could find was to uninstall and go back to an unmaintained, but much more reliable solution for now. This is a blocker, we need it to be 100% reliable.
This looks to be an issue with Karma: https://github.com/karma-runner/karma/issues/945
Unfortunately it seems that Karma is a dead project. That issue was opened in 2014 and was closed since no one worked on the issue
It is absolutely not dead! We work with limited resources, so we cannot address every issue immediately and sadly that one got closed a long time ago; I will be adding it to my working queue. Also, your collaboration is welcome.
I am experiencing the same issue after addressing the following message from our test run:
'karma-coverage-istanbul-reporter' usage has been deprecated since version 11.
Please install 'karma-coverage' and update 'karma.conf.js.' For more info, see https://github.com/karma-runner/karma-coverage/blob/master/README.md
I am going to revert the change in my project, and follow this issue, in hopes it gets resolved soon.
Some additional observations: I have seen the coverage report generated one time successfully, and if I scope the files in my test runner to a smaller subset of .spec.ts
files, it is generated every time. This clearly seems to be a memory or cpu issue. Whats troubling to me is that the process exits as though it successfully generated the report!
Update: Reverting back to karma-coverage-istanbul-reporter 100% fixed the issue, and our coverage reports are being reliably generated again, despite the deprecation warning. I can confirm there are issues with code coverage reports and:
Angular:
$ ng --version
_ _ ____ _ ___
/ \ _ __ __ _ _ _| | __ _ _ __ / ___| | |_ _|
/ △ \ | '_ \ / _` | | | | |/ _` | '__| | | | | | |
/ ___ \| | | | (_| | |_| | | (_| | | | |___| |___ | |
/_/ \_\_| |_|\__, |\__,_|_|\__,_|_| \____|_____|___|
|___/
Angular CLI: 12.2.1
Node: 14.15.5
Package Manager: npm 6.14.11
OS: darwin x64
Angular: 12.2.1
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, material, material-moment-adapter
... platform-browser, platform-browser-dynamic, platform-server
... router
Package Version
---------------------------------------------------------
@angular-devkit/architect 0.1202.1
@angular-devkit/build-angular 12.2.1
@angular-devkit/core 12.2.1
@angular-devkit/schematics 12.2.1
@angular/flex-layout 12.0.0-beta.34
@schematics/angular 12.2.1
rxjs 6.6.7
typescript 4.3.5
webpack 5.50.0
karma stuffs from package.json:
$ cat package.json | grep karma
"karma-spec-reporter": "^0.0.32",
"karma": "~6.3.4",
"karma-chai": "^0.1.0",
"karma-chrome-launcher": "~3.1.0",
"karma-cli": "^2.0.0",
"karma-coverage": "~2.0.3",
"karma-jasmine": "~4.0.0",
"karma-jasmine-html-reporter": "^1.5.0",
"karma-junit-reporter": "^2.0.1",
"karma-phantomjs-launcher": "^1.0.4",
"karma-typescript": "^5.5.1",
I hope this helps
I think I am seeing this problem as well. Intermittently, the coverage directory isn't created. I don't know that my project is large or not... not sure of the baseline!
$ ./node_modules/.bin/cloc --include-lang=TypeScript src/app/ vendored/
381 text files.
380 unique files.
148 files ignored.
github.com/AlDanial/cloc v 1.90 T=1.48 s (165.3 files/s, 117159.1 lines/s)
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
TypeScript 245 2320 1451 169847
-------------------------------------------------------------------------------
SUM: 245 2320 1451 169847
-------------------------------------------------------------------------------
An example of failure:
✓ should submit form
✓ should submit form API data
✓ should check permission for edit
✓ should edit & delete API key
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 0 (skipped 1) SUCCESS (59.589 secs / 17.681 secs)
TOTAL: 182 SUCCESS
____ _ _ ____ ____ _____ ____ ____
/ ___|| | | |/ ___/ ___| ____/ ___/ ___|
\___ \| | | | | | | | _| \___ \___ \
___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/
$ ls -la coverage/
ls: cannot access coverage/: No such file or directory
Cleaning up file based variables
An example of success:
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
TOTAL: 182 SUCCESS
Chrome Headless 94.0.4606.61 (Linux x86_64): Executed 182 of 183 (skipped 1) SUCCESS (17.465 secs / 15.649 secs)
TOTAL: 182 SUCCESS
08 10 2021 11:15:39.085:WARN [launcher]: ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
____ _ _ ____ ____ _____ ____ ____
/ ___|| | | |/ ___/ ___| ____/ ___/ ___|
\___ \| | | | | | | | _| \___ \___ \
___) | |_| | |__| |___| |___ ___) |__) |
|____/ \___/ \____\____|_____|____/____/
$ ls -la coverage/
total 832
drwxrwxr-x 4 gitlab-runner gitlab-runner 152 Oct 8 11:15 .
drwxrwxr-x 8 gitlab-runner gitlab-runner 4096 Oct 8 11:15 ..
-rw-rw-r-- 1 gitlab-runner gitlab-runner 658117 Oct 8 11:15 cobertura.txt
drwxrwxr-x 3 gitlab-runner gitlab-runner 182 Oct 8 11:15 report-html
drwxrwxr-x 3 gitlab-runner gitlab-runner 42 Oct 8 11:15 report-lcov
-rw-rw-r-- 1 gitlab-runner gitlab-runner 160020 Oct 8 11:15 report-lcovonly.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 697 Oct 8 11:15 teamcity.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 302 Oct 8 11:15 text-summary.txt
-rw-rw-r-- 1 gitlab-runner gitlab-runner 14499 Oct 8 11:15 text.txt
What is kind of interesting is that I get the SIGKILL in the case in which it works.... but maybe that's just karma killing the browser as it writes out the reports.
I also had no problems when using Istanbul previously.
Due to the warning, I'm just going to arbitrarily try setting the timeout higher in karma.conf.js and see what happens... I can't justify why it might help, but I'd like to see the warning go away anyway:
processKillTimeout: 60 * 1000
update: it didn't help. I still get intermittent failures to create the coverage output. update 2: reducing the number of kinds of reports to just text/text-summary might have helped. Waiting for more data. update 3: seems to still happen. Maybe when cicd machine is under higher load. Will need to go back to istanbul version more than likely.
I have exactly the same issue. The coverage reports are generated only when the following messasge is logged:
ChromeHeadless was not killed in 2000 ms, sending SIGKILL.
Is there a way to force karma to always send this? Or any other updates on this issue?
From a quick look it seems that the problem is that onRunComplete
returns a Promise, but Karma does not wait for for it. As a result Karma shutdown will race with the writing of the coverage report.
Unfortunately, without a reproduction it is hard to confirm or reject the theory. I wonder if somebody having the problem can run tests using this branch and see if it solves the problem? The only change there is that Karma will not call process.exit
once done, thus allowing all async activities to complete.
npm i devoto13/karma#test-coverage
UPD Looks like that promise is actually awaited in the onExit
callback. Another place, which is not synchronized is this one 🤔 I think it still will be useful to perform the test, so we can confirm that the issue is caused by the race and focus on locating and fixing this race.
I wonder if somebody having the problem can run tests using this branch and see if it solves the problem?
Trying now. I'll do a few runs on this and report back to see how it does with our code. Thanks
First run worked. Our CI server is busy today and the tests can take a while, so I'll run a few more runs on the branch and see if it is working 100% of the time across several runs.
I tried the branch that @devoto13 posted and it does seem to fix the problem 👍 Out of 20 executions the coverage report was generated in all of them.
Thanks for the confirmation @ozkoidi! So it seems that indeed the problem is the race between the Karma shutdown and the writing of the coverage. So far this looks like the main offender. I'll try to put a PR to properly wait for it later today.
I did not have any reproductions of the issue during my checks on the branch. We are going to try to use it for all our code until it is released and I'll report back if anyone fails to have coverage generated. Thank you!
@arobinson The branch was only intended for testing the race, I think it will always exit with 0 exit code (even when tests have failed).
Haven't found any more races, so hopefully, https://github.com/karma-runner/karma-coverage/pull/463 is the final fix. You can also give it a try on your codebase by installing it from the branch using the below command:
$ npm i devoto13/karma-coverage#fix-race
I think it will always exit with 0 exit code
FYI, I got an exit code of 1 with a test failure even with that branch
Using the fix-race branch for https://github.com/karma-runner/karma-coverage/pull/463, I tried it on our software and the build failed:
Test report file /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/junit/TESTS-Chrome_Headless_92.0.4515.159_(Linux_x86_64).xml was length 0
The tests started fine:
[2022-01-20T16:15:59.383Z] - Generating browser application bundles (phase: setup)...
[2022-01-20T16:16:01.932Z] Test results are being saved to: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results
[2022-01-20T16:16:01.932Z] 20 01 2022 16:16:01.915:INFO [karma-server]: Writing browser console to file: /tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/console.log
[2022-01-20T16:16:01.932Z]
[2022-01-20T16:16:01.932Z] START:
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.938:INFO [karma-server]: Karma v6.3.11 server started at http://localhost:9876/
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.939:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
[2022-01-20T16:16:02.188Z] 20 01 2022 16:16:01.949:INFO [launcher]: Starting browser Chrome
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:05.217:WARN [launcher]: Chrome has not captured in 120000 ms, killing.
[2022-01-20T16:18:08.605Z] 20 01 2022 16:18:07.315:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:18:20.800Z] 20 01 2022 16:18:19.493:INFO [launcher]: Trying to start Chrome again (1/2).
[2022-01-20T16:19:42.190Z] ✔ Browser application bundle generation complete.
[2022-01-20T16:19:42.190Z] 20 01 2022 16:19:33.627:INFO [Chrome Headless 92.0.4515.159 (Linux x86_64)]: Connected on socket uvg6glKjq65iOVWJAAAB with id 95294800
[2022-01-20T16:20:04.076Z] AppComponent Test Suite
[2022-01-20T16:20:04.076Z] ✔ Initial Rendering
But at the end, the tests failed badly:
[2022-01-20T16:31:58.196Z] ✔ 7564 tests completed
[2022-01-20T16:31:58.196Z] ℹ 35 tests skipped
[2022-01-20T16:32:13.033Z] 20 01 2022 16:32:12.590:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.792:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.794:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: UnhandledRejection: Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] 20 01 2022 16:32:13.795:ERROR [karma-server]: [Error: EEXIST: file already exists, mkdir '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'] {
[2022-01-20T16:32:13.968Z] errno: -17,
[2022-01-20T16:32:13.968Z] code: 'EEXIST',
[2022-01-20T16:32:13.968Z] syscall: 'mkdir',
[2022-01-20T16:32:13.968Z] path: '/tmp/workspace/Client/nw-client-sonar-pull-request/development/nw-client/client-source/test-results/coverage'
[2022-01-20T16:32:13.968Z] }
That EEXIST
error was repeated 34,494 times in the job log before Jenkins stopped recording the output. Looks like it got caught in an infinite loop
Thanks for the feedback @arobinson. Looks like there is still a race between a check for a directory existence and creation in the helper, which is executed concurrently for all reporters. I'll look into fixing it.
Interesting that it got into an infinite loop though 🤔
Here is another attempt to fix this:
$ npm i devoto13/karma-coverage#fix-race
$ npm i devoto13/karma#fix-helper
which should address the infinite loop and the race when creating the same directory concurrently. I've tried it on the fresh Angular project and it seems to work now.
@alan-agius4
I've noticed since upgrading our project to Angular 13 that, when running ng test
, the karma-coverage-instanbul-reporter
deprecation warning has gone away.
Do you know if this was this intentional? I combing through the v13 commits to angular-cli, I could not find where it was changed. I thought you might have some insight.
Yea, that’s intentional as we no longer support it.
that said, you can still use it, if you provide the reporter configuration yourself.
Testing the latest patches, I did get code coverage numbers without any issues. ~I'll try a couple of more times to make sure~
Did 3 runs, all collected coverage correctly
:tada: This issue has been resolved in version 2.1.1 :tada:
The release is available on:
Your semantic-release bot :package::rocket:
I upgraded and I believe I'm still not getting a coverage directory in cicd. Locally, yes, but in cicd, no. The main difference being the load being higher on the cicd machine from other jobs running in parallel.
$ grep karma-coverage yarn.lock
karma-coverage-istanbul-reporter@^3.0.3:
resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage-istanbul-reporter/-/karma-coverage-istanbul-reporter-3.0.3.tgz#f3b5303553aadc8e681d40d360dfdc19bc7e9fe9"
karma-coverage@^2.1.1:
resolved "http://build-artifactory.eng.vmware.com/artifactory/api/npm/npm/karma-coverage/-/karma-coverage-2.1.1.tgz#7434f8736841e08de3e6c302632e8d239634f388"
(I'll also try without the istanbul one also in node_modules)
$ more karma.conf.js
const path = require("path")
const fs = require("fs")
function get_backend_url() {
const config_file = path.resolve(process.cwd(), ".karma-backend-config.json")
try {
const data = fs.readFileSync(config_file)
const config = JSON.parse(data)
return config.LIVE_SERVER_URL
} catch (error) {
return "http://localhost:12345"
}
}
const backend_url = get_backend_url()
module.exports = function (config) {
config.set({
failOnSkippedTests: false,
basePath: "",
frameworks: ["jasmine", "@angular-devkit/build-angular"],
processKillTimeout: 60 * 1000,
plugins: [
require("karma-jasmine"),
require("karma-chrome-launcher"),
require("karma-coverage"),
require("@angular-devkit/build-angular/plugins/karma"),
require("karma-spec-reporter"),
],
logLevel: config.LOG_INFO,
client: {
clearContext: false, // leave Jasmine Spec Runner output visible in browser
jasmine: {
random: false,
timeoutInterval: 300000,
},
},
// doesn't always work: https://github.com/karma-runner/karma-coverage/issues/434
coverageReporter: {
dir: path.join(__dirname, "coverage"),
reporters: [
{ type: "html", subdir: "report-html" },
{ type: "lcov", subdir: "report-lcov" },
{ type: "text-summary", subdir: ".", file: "text-summary.txt" },
],
fixWebpackSourcePaths: true,
},
// https://stackoverflow.com/questions/54744584/karma-disconnected-because-no-message-in-10000-ms
captureTimeout: 300000,
browserDisconnectTolerance: 3,
browserDisconnectTimeout: 300000,
browserNoActivityTimeout: 300000,
reporters: [
"progress",
"spec",
"coverage",
],
specReporter: {
suppressSkipped: false,
},
hostname: "0.0.0.0",
port: 9876,
colors: true,
autoWatch: true,
browsers: ["ChromeHeadless"],
singleRun: false,
restartOnFileChange: true,
proxies: {
"/api/": backend_url + "/api/",
"/ajax/": backend_url + "/ajax/",
"/saml2/": backend_url + "/saml2/",
},
})
}
@rrauenza Are you also using karma 6.3.15 which contains the second part of the fix?
If so I wonder if this can be related to the issue reported in https://github.com/karma-runner/karma/issues/3745, where onBrowserStart
is not called. Would you be willing to add some logs to the on*
hooks in https://github.com/karma-runner/karma-coverage/blob/9e8492c78dd92a5b7d906d82001104c42d4f9a7e/lib/reporter.js#L189 and run the same code locally and in your CI to pinpoint what exactly is not executed/executed differently? I think onBrowserStart
should be a good place to start. With this information, I may be able to pinpoint the issue and make the fix.
The thing with onBrowserStart
is that it is sent over the socket.io connection by the browser and I suspect that socket.io may lose messages while upgrading to the WebSocket connection when the host/browser is under heavy load. There were several issues reported that seem related, but it is tricky to reproduce and confirm from our side.
@arobinson I wonder if you had time to upgrade karma
and karma-coverage
in your project and can confirm whether the flaky coverage issues was resolved or it is still flaky for you?
@rrauenza Are you also using karma 6.3.15 which contains the second part of the fix?
No! I have 6.3.4 according to my yarn.lock. I'll upgrade. Does karma-coverage not require the newer version in its dependency requirements?
No, karma plugins are not used to have peerDependencies, that's something I would like to better address in the future.
Unfortunately the code coverage is often not generated for me when the unit tests run on a Azure Devops pipeline (always works on local). I'm using the latest packages:
Here is a comparison of the end of the logs in case it helps. Since the source code is the same on both executions, I wonder if the difference in the order of the logged actions is the cause of the issue.
With test coverage generated 👍
Without test coverage: 👎
@ozkoidi From your logs, you seem to be using the https://github.com/joeljeske/karma-parallel plugin which may be the reason. Can you please try to run without it and see if you still get the same problem?
I'm not familiar with that plugin and will try to look at it when I have time to see if it does something suspicious.
@devoto13, thanks for your suggestion. I removed the karma-parallel package and the result is still the same, so no need for you to look into that package.
In case it helps the specs of the Microsoft hosted agents running the tests are: 2 core CPU, 7 GB of RAM, and 14 GB of SSD disk space. I think those VM should be enough to run the test properly but I don't know what else could make the test not generate coverage reports randomly. Updated logs:
With test coverage generated 👍
Without test coverage 👎
@ozkoidi Thank you for the quick feedback. Can you please install karma
and karma-coverage
from these two branches and make more sample runs on your CI?
$ npm i devoto13/karma#test-coverage
$ npm i devoto13/karma-coverate#test-coverage
The first one removes the call to the process.exit
on Karma shut down, so if the problem is resolved by the change, it means that we still have a race somewhere in the coverage generation routine. Per https://github.com/karma-runner/karma-coverage/issues/434#issuecomment-1013479221. Here is the code.
The second one is based on https://github.com/karma-runner/karma-coverage/issues/434#issuecomment-1033808800. It adds logs to all karma-coverage
hooks so that we can compare good and bad runs and see if we can get some clue about what is different. Here is the code.
Hi there,
This library is now recommended for code coverage when using v11 of the angular-cli. We are currently using karma-coverage-istanbul-reporter because we have yet to get successful coverage reports with karma-coverage. Seeing, however, that there are deprecation warnings when using the istanbul reporter library with the v11 angular-cli, we would very much like to move off it.
Perhaps someone here can help discern what is wrong with our configuration or point us in the right direction.
I will include the karma.config, and test output debug logs for two different projects.
The first project (PROJECT-1) is a bare project that was initialized with
ng new
command. It was updated to Angular 11 using theng update
command from angular-cli. The point of including this project is to serve as a 'control' for the second project.The second project (PROJECT-2) comes from a relatively large production codebase. In this codebase is roughly 5000 unit tests.
Important points
PROJECT-1 tests are run with:
ng test
. PROJECT-2 tests are run with:node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng test
.--max_old_space_size
configuration property mentioned about.Below is the karma config file shared by the two projects and the debug logs that are output when running tests.
Karma config
PROJECT-1-debug-output.txt PROJECT-2-debug-output.txt
In the PROJECT-2 output you will see only ~20 tests ran. This is because I used an
fdescribe
to run a single test file so as to reduce the noise a little. The same result--no coverage--happens with or without this change.The point of failure seems to be here:
Thanks in advance for your support.