Our team utilizes GitHub Actions to run a series of TestCafe tests nightly against our environments. During random runs; not often, but potentially once a month after 10+ runs a day; the test run will freeze, resulting in GHA force closing the runner after 6 hours. Investigating into this, we haven't been able to fully reproduce this issue, outside of manually closing the browser during a test. Our tests on GHA (ubuntu:latest) utilize Chrome in Headless mode, and our tests when developing (Windows 11) use Chrome standard (some tests using --guest), but not headless. We aren't yet convinced the browser is closing during the GHA tests, but this is our current/best lead.
What is the Current behavior?
During our tests in Windows, closing the browser mid session (once the fixture/test start their actions), calls to await t functions will hang, as they cannot be completed.
What is the Expected behavior?
Ideally, the closed browser / lost connection, should be detected as a failure, and either one of the following actions:
Quit the run entirely
Attempt to run the fixture 'after' function, and abort the rest of the queued tests
Likely, attempting to run the test/fixture afterEach can't run, due to the lack of t environment anymore.
Reporters should be able to finalize their run and export their results of the tests that [did] run.
What is the public URL of the test page? (attach your complete example)
Can be repro'd on example.com
What is your TestCafe test code?
runner.js
const fs = require('fs');
const path = require('path');
const createTestCafe = require('testcafe');
/* Required for tests that spawn child tests */
global.createTestCafe = createTestCafe;
/* Required for running via proxy, and fake media tests */
const selfSigned = require('openssl-self-signed-certificate');
const sslOptions = {
key: selfSigned.key,
cert: selfSigned.cert
};
(async () => {
const testcafe = await createTestCafe({
hostname: 'localhost',
sslOptions
});
const browsers = [];
let chrome = [];
let runnerOptions = {};
if(process.platform==='win32') { // windows
chrome.push('chrome --start-maximized');
} else { // not windows / github actions likely
chrome.push('chrome:headless');
runnerOptions = {
quarantineMode: {
attemptLimit: 3,
successThreshold: 1
}
}
}
chrome.push('--allow-insecure-localhost');
/* If NODE_MEDIA is set, require fake media sources in tests */
/* Optional: Set NODE_AUDIO or NODE_VIDEO to files in ~/assets, to override the default files */
if(process.env.NODE_MEDIA) {
// check if media files exist
let audioFile = path.resolve(`assets/${process.env.NODE_AUDIO ?? 'source_audio.wav'}`);
let videoFile = path.resolve(`assets/${process.env.NODE_VIDEO ?? 'source_video2.y4m'}`);
// add if exists
if(fs.existsSync(audioFile) && fs.existsSync(videoFile)) {
console.log("Using Chrome with fake video and audio");
chrome.push('--use-fake-ui-for-media-stream');
chrome.push('--use-fake-device-for-media-stream');
chrome.push(`--use-file-for-fake-video-capture=${videoFile}`);
chrome.push(`--use-file-for-fake-audio-capture=${audioFile}`)
}
}
chrome = chrome.join(' ');
let failedCount = 0;
try {
function recurringLogger(x, ms) {
let onceLock = false;
const logIt = async () => {
if(onceLock) return;
onceLock = true;
let openSession = x.proxy?.openSessions.keys().next().value;
if(openSession) {
let session = x.proxy.openSessions.get(openSession);
let sessionCount = session.proxy.server1._connections
console.log(`Connection count: ${sessionCount}`);
if(sessionCount===0) {
console.log("Oh no, browser closed?!?!? Stopping");
await runner.stop();
console.log("Stop finished?");
failedCount.cancel();
console.log("Cancel called");
await testcafe.close();
console.log("Testcafe closed?");
}
} else {
console.log("No session open");
}
onceLock = false;
// console.log(x.proxy?.openSessions);
}
let loop = setInterval(logIt, ms);
process.on('beforeExit', () => {
console.log("Final exit thoughts");
clearInterval(loop);
})
}
const runner = testcafe.createRunner();
const testSources = [
'tests/**/*.js'
];
failedCount = runner
.src(testSources)
.browsers(chrome)
.concurrency(1)
.video('failures/videos', {failedOnly: true})
.run(runnerOptions);
recurringLogger(runner, 500);
failedCount = await failedCount;
console.log('Tests failed: ' + failedCount);
await testcafe.close();
}
catch(e){
console.log('Error in running tests',e);
}
finally {
process.exit(failedCount);
}
})();
test.js
import { Selector } from 'testcafe';
fixture.only('Fixture')
.page('http://example.com')
;
test('test', async t => {
for(let i = 0; i < 20; i++) {
await t
.expect(Selector('h1').innerText).eql('Example Domain')
.wait(1000);
console.log("Completed check: ", i);
}
});
Your complete configuration file
No response
Your complete test report
No response
Screenshots
No response
Steps to Reproduce
run node runner.js
note the logging showing the browser connection initializing, becoming available, and the test counter running
close the browser before the test finishes, but after the counting started; within 20 seconds of the test run
TestCafe version
3.6.2
Node.js version
v16.20.2
Command-line arguments
node runner.js
Browser name(s) and version(s)
Chrome 128
Platform(s) and version(s)
No response
Other
This test has been done with both Native Automation enabled and disabled with no difference.
We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.
What is your Scenario?
Our team utilizes GitHub Actions to run a series of TestCafe tests nightly against our environments. During random runs; not often, but potentially once a month after 10+ runs a day; the test run will freeze, resulting in GHA force closing the runner after 6 hours. Investigating into this, we haven't been able to fully reproduce this issue, outside of manually closing the browser during a test. Our tests on GHA (ubuntu:latest) utilize Chrome in Headless mode, and our tests when developing (Windows 11) use Chrome standard (some tests using --guest), but not headless. We aren't yet convinced the browser is closing during the GHA tests, but this is our current/best lead.
What is the Current behavior?
During our tests in Windows, closing the browser mid session (once the fixture/test start their actions), calls to
await t
functions will hang, as they cannot be completed.What is the Expected behavior?
Ideally, the closed browser / lost connection, should be detected as a failure, and either one of the following actions:
afterEach
can't run, due to the lack oft
environment anymore.Reporters should be able to finalize their run and export their results of the tests that [did] run.
What is the public URL of the test page? (attach your complete example)
Can be repro'd on example.com
What is your TestCafe test code?
runner.js
test.js
Your complete configuration file
No response
Your complete test report
No response
Screenshots
No response
Steps to Reproduce
node runner.js
TestCafe version
3.6.2
Node.js version
v16.20.2
Command-line arguments
node runner.js
Browser name(s) and version(s)
Chrome 128
Platform(s) and version(s)
No response
Other
This test has been done with both Native Automation enabled and disabled with no difference.