Question: Error while cucumber report generation post test execution

gitToSantosh commented 2 days ago

I often get below error post test case(s) execution, while generation of cucumber report. The json report file also remains empty.

Error in reporter Error: Pickle not found
    at TestCase.findPickle (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\TestCase.ts:133:24)
    at TestCase.addRun (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\TestCase.ts:52:27)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:123:16
    at Array.forEach (<anonymous>)
    at MessagesBuilder.createTestCases (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:114:23)
    at MessagesBuilder.doBuildMessages (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:91:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at CucumberReporterAdapter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\index.ts:69:5)  
    at ReporterV2Wrapper.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\reporterV2.js:91:12)
    at wrapAsync (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:79:12)
    at Multiplexer.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:51:25)
    at InternalReporter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\internalReporter.js:70:12)
    at Runner.runAllTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\runner\runner.js:77:28)
    at runTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:218:18)
    at t.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:55:7)

Error in reporter Error: Pickle not found
    at TestCase.findPickle (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\TestCase.ts:133:24)
    at TestCase.addRun (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\TestCase.ts:52:27)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:123:16
    at Array.forEach (<anonymous>)
    at MessagesBuilder.createTestCases (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:114:23)
    at MessagesBuilder.doBuildMessages (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:91:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at CucumberReporterAdapter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\index.ts:69:5)  
    at ReporterV2Wrapper.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\reporterV2.js:91:12)
    at wrapAsync (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:79:12)
    at Multiplexer.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:51:25)
    at InternalReporter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\internalReporter.js:70:12)
    at Runner.runAllTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\runner\runner.js:77:28)
    at runTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:218:18)
    at t.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:55:7)

vitalets commented 2 days ago

I will definitely enrich error message with more details. In upcoming playwright-bdd v8 this part was changed, so the error should gone. Is that error appears on every test run or it's a floating one?

gitToSantosh commented 2 days ago

I think I am still using 7.5.0. It does not appear on every test though. But often when I run multiple tests I see this, have no idea why it happens

vitalets commented 2 days ago

I think I am still using 7.5.0.

Yes, v8 is not released yet.

It does not appear on every test though.

Could you show the content of your playwright.config.ts (if possible)?

gitToSantosh commented 2 days ago

Sometimes I also see below error.

Error in reporter Error: Attachment without body and url
    at toEmbeddedAttachment (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\attachments\external.ts:16:30)
    at HtmlReporter.handleAttachment (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\html.ts:87:29)
    at EventEmitter.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\html.ts:59:14)
    at EventEmitter.emit (node:events:519:28)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:109:54
    at Array.forEach (<anonymous>)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:109:16
    at Array.forEach (<anonymous>)
    at MessagesBuilder.emitMessages (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:106:32)
    at CucumberReporterAdapter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\index.ts:70:37)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ReporterV2Wrapper.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\reporterV2.js:91:12)
    at wrapAsync (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:79:12)
    at Multiplexer.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:51:25)
    at InternalReporter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\internalReporter.js:70:12)
    at Runner.runAllTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\runner\runner.js:77:28)
    at runTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:218:18)
    at t.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:55:7)
Error in reporter Error: Attachment without body and url
    at toEmbeddedAttachment (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\attachments\external.ts:16:30)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:342:56
    at Array.map (<anonymous>)
    at JsonReporter.getStepData (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:341:44)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:199:23
    at Array.map (<anonymous>)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:197:58
    at Array.map (<anonymous>)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:193:30
    at Array.map (<anonymous>)
    at JsonReporter.onTestRunFinished (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:186:59)
    at EventEmitter.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\json.ts:136:14)
    at EventEmitter.emit (node:events:519:28)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:109:54
    at Array.forEach (<anonymous>)
    at C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:109:16
    at Array.forEach (<anonymous>)
    at MessagesBuilder.emitMessages (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\messagesBuilder\index.ts:106:32)
    at CucumberReporterAdapter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright-bdd\src\reporter\cucumber\index.ts:70:37)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at ReporterV2Wrapper.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\reporterV2.js:91:12)
    at wrapAsync (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:79:12)
    at Multiplexer.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\multiplexer.js:51:25)
    at InternalReporter.onEnd (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\reporters\internalReporter.js:70:12)
    at Runner.runAllTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\runner\runner.js:77:28)
    at runTests (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:218:18)
    at t.<anonymous> (C:\Users\ADMIN\Desktop\sample-automation\node_modules\playwright\lib\program.js:55:7)

here is my playwright config file

/* eslint-disable no-undef */
import { defineConfig } from '@playwright/test';
import { defineBddConfig, cucumberReporter } from 'playwright-bdd';
import dotenv from 'dotenv';
import type { GitHubActionOptions } from '@estruyf/github-actions-reporter';

// Load environment variables from the root .env file
dotenv.config( { path: './.env' } );

// Distinguish shard runs from regular local runs and merge-reports run
const isShardRun = process.argv.some( ( a ) => a.startsWith( '--shard' ) );
const isCiRun = process.env.CI === 'true' ? true : false;
const isLogVideo = process.env.LOG_VIDEO === 'true' ? true : false;
const isE2ERun = process.env.IS_E2E_RUN === 'true' ? true : false;

const testDir = defineBddConfig( {
    features: 'tests/features/**/*.feature',
    steps: [ 'tests/steps/*.ts', 'tests/fixtures/*.ts' ],
    tags: process.env.TAGS,
} );

export default defineConfig( {
    name: `Execute tests ${ process.env.MARKET } ${ process.env.ENV }`,
    globalSetup: require.resolve( './tests/fixtures/global-setup' ),
    testDir,
    // testDir: './tests',
    outputDir: './reports/test-results',
    preserveOutput: 'always',
    maxFailures: isE2ERun ? 1 : 3,
    fullyParallel: false,
    workers: 1,
    timeout: 50 * 60 * 1000,
    globalTimeout: 50 * 60 * 1000,
    expect: {
        timeout: isCiRun ? 30_000 : 60_000,
    },
    reporter: isShardRun
        ? 'blob'
        : [
            cucumberReporter( 'html', {
                outputFile: 'reports/cucumber-report/index.html',
                externalAttachments: false,
                // attachmentsBaseURL: 'http://127.0.0.1:8080/data',
                // skipAttachments: [ 'video/webm', 'application/zip' ],
            } ),
            cucumberReporter( 'json', {
                outputFile: 'reports/cucumber-report/report.json',
                // skipAttachments: [ 'video/webm', 'application/zip' ],
            } ),
            [ 'html', { open: 'always', outputFolder: 'reports/playwright-report' } ],
            // [ 'html', { open: 'always', outputFolder: 'playwright-report' } ],
            [ 'list', { printSteps: true } ],
            // [ './tests/fixtures/myCustomReporter.ts' ],
            // [ '@estruyf/github-actions-reporter', <GitHubActionOptions> {
            //  title: `Execute tests ${ process.env.MARKET } ${ process.env.ENV }`,
            //  showError: true,
            //  showAnnotations: false,
            //  // useDetails: true,
            //  includeResults: [ 'pass', 'skipped', 'fail', 'flaky' ]
            // } ],
            // [ 'monocart-reporter', {
            //  name: 'My Test Report',
            //  outputFile: 'reports/monocart-report/index.html',
            // } ]
        ],
    use: {
        actionTimeout: 30_000,
        // navigationTimeout: 30_000,
        navigationTimeout: 60_000,
        launchOptions: {
            args: [ '--start-maximized' ],
        },
        screenshot: 'on',
        trace: 'on',
        video: {
            mode: isLogVideo ? 'on' : 'off',
            size: { height: 720, width: 1280 }
        }
    },
    projects: [
        {
            // name: 'chromium',
            use: {
                viewport: isCiRun ? { width: 1920, height: 1080 } : null,
            },
        },

    ],
} );

gitToSantosh commented 9 hours ago

When will the v8 get released ??

Also , any idea why we see the other error posted above ??
Error in reporter Error: Attachment without body and url

vitalets commented 6 hours ago

When will the v8 get released ??

Within this or next week.

Also , any idea why we see the other error posted above ?? Error in reporter Error: Attachment without body and url

Do you add any attachments manually in tests?

gitToSantosh commented 5 hours ago

Yes I am adding attachments manually, (2 attachments, hence twice the error ? )

This is within my custom fixture, I use testInfo to attach to the report. I used both testInfo and $testInfo but i get same error

await $testInfo.attach( 'logs.txt', {
            body: logsContent,
            contentType: 'text/plain'
        } );

await testInfo.attach( 'logs.txt', {
            body: logsContent,
            contentType: 'text/plain'
        } );

gitToSantosh commented 4 hours ago

Nope, even when I commented out the attachments, I still see the error twice , no idea why

gitToSantosh commented 3 hours ago

I think I am able to reproduce the error now .

I have jar file which i run in a child process. It takes an argument and return a string. Every time this piece of code runs, I get the error "Error in reporter Error: Attachment without body and url"

Then( 'I see in title {string}', async ( { page }, pageTitle: string ) => {
    // Verify the page title
    await expect( page ).toHaveTitle( pageTitle );

    try {
        // Run the child process command
        const output = execSync( 'java -jar library.jar "XXXX XXXX XXXX XXXX"' ).toString();

        // Log the output or perform further actions
        console.log( 'Java Help Output:', output );
    } catch ( error ) {
        // Handle potential errors
        console.error( 'Error running command:', error.message );
    }
} );

Also even When you run a simple command like "java -help" in a child process, you can see stderr attachment in playwright html report

vitalets / playwright-bdd

Question: Error while cucumber report generation post test execution #250