search

reportportal / agent-js-testcafe

Agent to integrate TestCafe with ReportPortal.
https://www.npmjs.com/package/@reportportal/testcafe-reporter-agent-js-testcafe
Apache License 2.0
4 stars 0 forks source link

Parallel testing fixture bleed #17

Open kflore16 opened 3 years ago

kflore16 commented 3 years ago

I have 3 files with a suite (fixture) in each file with 10 tests per file, the code snippet describes how they're written


fixture.disablePageCaching'TestA bleed 10 tests 1-10'//  The two other fixtures are TestB and TestC, 11-20 and 21-30 respectively
  .page`https://devexpress.github.io/testcafe/example/`.beforeEach( async t =>{

    await t.maximizeWindow()
  })
const Truth = true
const TestName = 'Test A'

//The Test numbering is in order with the fixture name
test.meta({
  smoke: 'yes',
  regression: 'yes',
  adhoc: 'yes',
  description: 'Test Descript for Test #1',
  attributes: [{ key: 'Type', value: 'Navigation' }],
})('Test 1 for ${TestName}', async t => {
  await t.expect(Truth).ok()
})`

Testcafe version 1.15.2 reporter-agent-js version 5.0.0 Report portal version API: 5.4.0, Index: 5.0.10 image The image shows that the tests are bleeding into the next fixture, all ran in chrome:headless with a -c 3 flag image Fixture A image Fixture B and similarly in Fixture C

``` PS C:\ npx testcafe -c 3 chrome:headless ./tests/DummyTest/*.test.js --skip-js-errors The "src" and "browsers" options from the configuration file will be ignored. Running tests in: - Chrome 94.0.4606.61 / Windows 10 Start launch sk56uj1cku66jswc Start test item sk56uj1cku66jswh Success start item sk56uj1cku66jswh Start test item sk56uj1cku66jswl Start test item sk56uj1cku66jswv Start test item sk56uj1cku66jsww Success start item sk56uj1cku66jsww Success start item sk56uj1cku66jswl Success start item sk56uj1cku66jswv √ TestA bleed 10 tests 1-10 - Test 1 for Test A Finish test item sk56uj1cku66jswl √ TestA bleed 10 tests 1-10 - Test 2 for Test A Finish test item sk56uj1cku66jswv √ TestA bleed 10 tests 1-10 - Test 3 for Test A Finish test item sk56uj1cku66jsww Start test item sk56uj1cku66ju61 Start test item sk56uj1cku66ju64 Start test item sk56uj1cku66ju67 Success start item sk56uj1cku66ju64 Success finish item sk56uj1cku66jswv Success finish item sk56uj1cku66jsww Success start item sk56uj1cku66ju61 Success finish item sk56uj1cku66jswl Success start item sk56uj1cku66ju67 √ TestA bleed 10 tests 1-10 - Test 4 for Test A Finish test item sk56uj1cku66ju61 √ TestA bleed 10 tests 1-10 - Test 5 for Test A Finish test item sk56uj1cku66ju64 Start test item sk56uj1cku66juyq √ TestA bleed 10 tests 1-10 - Test 6 for Test A Finish test item sk56uj1cku66ju67 Start test item sk56uj1cku66juyy Start test item sk56uj1cku66juz5 Success finish item sk56uj1cku66ju67 Success finish item sk56uj1cku66ju61 Success finish item sk56uj1cku66ju64 Success start item sk56uj1cku66juz5 Success start item sk56uj1cku66juyy Success start item sk56uj1cku66juyq √ TestA bleed 10 tests 1-10 - Test 7 for Test A Finish test item sk56uj1cku66juyq Start test item sk56uj1cku66jvr0 √ TestA bleed 10 tests 1-10 - Test 8 for Test A Finish test item sk56uj1cku66juyy √ TestA bleed 10 tests 1-10 - Test 9 for Test A Finish test item sk56uj1cku66juz5 Success finish item sk56uj1cku66juyq Start test item sk56uj1cku66jvrx Start test item sk56uj1cku66jvs3 Success start item sk56uj1cku66jvr0 Success finish item sk56uj1cku66juz5 Success start item sk56uj1cku66jvrx Success finish item sk56uj1cku66juyy Success start item sk56uj1cku66jvs3 √ TestA bleed 10 tests 1-10 - Test 10 for Test A Finish test item sk56uj1cku66jvr0 Start test item sk56uj1cku66jwix √ TestB bleed 10 tests 11-20 - Test 11 for Test B Finish test item sk56uj1cku66jvrx √ TestB bleed 10 tests 11-20 - Test 12 for Test B Finish test item sk56uj1cku66jvs3 Success finish item sk56uj1cku66jvr0 Success start item sk56uj1cku66jwix Start test item sk56uj1cku66jwj6 Start test item sk56uj1cku66jwjr Start test item sk56uj1cku66jwk5 Success finish item sk56uj1cku66jvrx Success start item sk56uj1cku66jwk5 Success finish item sk56uj1cku66jvs3 Success start item sk56uj1cku66jwj6 Success start item sk56uj1cku66jwjr √ TestB bleed 10 tests 11-20 - Test 13 for Test B Finish test item sk56uj1cku66jwj6 Start test item sk56uj1cku66jxbb √ TestB bleed 10 tests 11-20 - Test 14 for Test B Finish test item sk56uj1cku66jwjr Start test item sk56uj1cku66jxca Success start item sk56uj1cku66jxbb √ TestB bleed 10 tests 11-20 - Test 15 for Test B Finish test item sk56uj1cku66jwk5 Success finish item sk56uj1cku66jwj6 Start test item sk56uj1cku66jxcu Success start item sk56uj1cku66jxca Success finish item sk56uj1cku66jwjr Success finish item sk56uj1cku66jwk5 Success start item sk56uj1cku66jxcu √ TestB bleed 10 tests 11-20 - Test 16 for Test B Finish test item sk56uj1cku66jxbb Start test item sk56uj1cku66jy3q √ TestB bleed 10 tests 11-20 - Test 17 for Test B Finish test item sk56uj1cku66jxca Success finish item sk56uj1cku66jxbb Start test item sk56uj1cku66jy4m Success start item sk56uj1cku66jy3q √ TestB bleed 10 tests 11-20 - Test 18 for Test B Finish test item sk56uj1cku66jxcu Start test item sk56uj1cku66jy5k Success finish item sk56uj1cku66jxca Success start item sk56uj1cku66jy4m Success finish item sk56uj1cku66jxcu Success start item sk56uj1cku66jy5k √ TestB bleed 10 tests 11-20 - Test 19 for Test B Finish test item sk56uj1cku66jy3q Start test item sk56uj1cku66jyvw √ TestB bleed 10 tests 11-20 - Test 20 for Test B Finish test item sk56uj1cku66jy4m Start test item sk56uj1cku66jywi √ TestC bleed 10 tests 21-30 - Test 21 for Test C Finish test item sk56uj1cku66jy5k Success finish item sk56uj1cku66jy3q Success start item sk56uj1cku66jyvw Success start item sk56uj1cku66jywi Start test item sk56uj1cku66jywq Start test item sk56uj1cku66jyx4 Success finish item sk56uj1cku66jy5k Success finish item sk56uj1cku66jy4m Success start item sk56uj1cku66jyx4 Success start item sk56uj1cku66jywq √ TestC bleed 10 tests 21-30 - Test 22 for Test C Finish test item sk56uj1cku66jyvw Start test item sk56uj1cku66jzo1 √ TestC bleed 10 tests 21-30 - Test 23 for Test C Finish test item sk56uj1cku66jywq Success finish item sk56uj1cku66jyvw Start test item sk56uj1cku66jzop Success start item sk56uj1cku66jzo1 √ TestC bleed 10 tests 21-30 - Test 24 for Test C Finish test item sk56uj1cku66jyx4 Success finish item sk56uj1cku66jywq Success start item sk56uj1cku66jzop Start test item sk56uj1cku66jzpi Success finish item sk56uj1cku66jyx4 Success start item sk56uj1cku66jzpi √ TestC bleed 10 tests 21-30 - Test 25 for Test C Finish test item sk56uj1cku66jzo1 √ TestC bleed 10 tests 21-30 - Test 26 for Test C Finish test item sk56uj1cku66jzop Start test item sk56uj1cku66k0gh Start test item sk56uj1cku66k0gm Success finish item sk56uj1cku66jzo1 Success start item sk56uj1cku66k0gh Success finish item sk56uj1cku66jzop √ TestC bleed 10 tests 21-30 - Test 27 for Test C Finish test item sk56uj1cku66jzpi Success start item sk56uj1cku66k0gm Start test item sk56uj1cku66k0ho Success finish item sk56uj1cku66jzpi Success start item sk56uj1cku66k0ho √ TestC bleed 10 tests 21-30 - Test 28 for Test C Finish test item sk56uj1cku66k0gh √ TestC bleed 10 tests 21-30 - Test 29 for Test C Finish test item sk56uj1cku66k0gm Success finish item sk56uj1cku66k0gh Success finish item sk56uj1cku66k0gm √ TestC bleed 10 tests 21-30 - Test 30 for Test C Finish test item sk56uj1cku66k0ho 30 passed (11s) Finish test item sk56uj1cku66jswh Finish test item sk56uj1cku66jwix Success finish item sk56uj1cku66k0ho Finish test item sk56uj1cku66jywi Success finish item sk56uj1cku66jwix Success finish item sk56uj1cku66jswh Success finish item sk56uj1cku66jywi Finish launch sk56uj1cku66jswc Success finish launch sk56uj1cku66jswc```
kflore16 commented 2 years ago

A similar behavior is observed with ReportingApi. using concurrency, The logs for an earlier test is delayed by X tests for X amount of browser instances.

kflore16 commented 2 years ago

TL:DR Tests are being completed in the wrong fixture when using concurrent runners. (Test A 1-10, Test B 11-20, and Test C 21-30 -> Test A completes till 1-12, Test B completes with 13-22, Test C completes 23-30 with 3 concurrent test runners) If I were to fix this myself, where would I start? @AmsterGet

AmsterGet commented 2 years ago

Hello @kflore16 ! As I understand it correctly, events from tests fired using the ReportingAPI are isolated in the process in which the tests are executed, so in this case we need a mechanism to synchronize data between processes during test execution. We already plan to implement this feature, but not in the near future. Firstly, I would try to solve this using testcafe and its reporting system, in case of failure, I think a more custom solution will be needed here.