[Feature]: [Custom Reporters] Custom IPC communication between runner process and main process possible?

kbrueckner commented 4 months ago

🚀 Feature Request

During runtime the main process spawns worker processes to run the actual tests. A solution is required which allows to use the playwright internal IPC communication between runner and main to forward custom messages which can be listened to when building custom reporters. The current implementation and interface options are not sufficient or rather more lack those capabilities.

Example

No response

Motivation

I am maintainer of a library which wraps playwright into screenplay pattern. (https://www.npmjs.com/package/@testla/screenplay-playwright) Since playwright is part of the absolute core of the lib, I wanted to use its reporter capabilities to build an own report which represents all the screenplay steps executed during runtime. This may look like the following example:

2024-07-01T11:00:38.144Z  [EXEC]  ↪ Andy Authorized attemptsTo Login.toApp  (restrictions.spec.ts:91:19)
2024-07-01T11:00:38.145Z  [EXEC]      ↪ Andy Authorized attemptsTo Navigate.to(url: '/')  (Login.ts:26:22)
2024-07-01T11:00:38.607Z  [DONE]      ↪ Andy Authorized attemptsTo Navigate.to(url: '/')  (Login.ts:26:22)

As per the Reporter interface which my custom reporter is built on I have access to stdout - so my current solution is build on stdout messages which I extract and process. Since printing to stdout is slowing down the execution quite a bit and spams a lot of messages I need an easy way to access the playwrights internal IPC to utilize it to forward custom events from worker to main so that I can get rid of the stdout solution.

mxschmitt commented 4 months ago

We have multiple ways of communicating between our workers and the reporters as of today:

Stdout / Stderr: You seem to already using it as of today. You can e.g. filter out output there if you users are only using your reporter. Attachments: There you can attach files to a test. Annotations: There you can attach custom data to a test. Or wrap things inside steps: https://playwright.dev/docs/api/class-test#test-step for making things more readable across the tooling e.g. trace viewer / ui mode. Also streaming into a file per worker could be a possible solution.

Marking it as a feature request for now

kbrueckner commented 4 months ago

It might be that i do not fully understand the details of attachments, annotations and the steps. But as per my understanding they would need to be put into the test code. This is not what I would like to achieve, I want to just run a defined test as is and react in the reporter on custom events. Currently the implementation via stdout works but has the mentioned caveats.

I would appreciate to have something similar like onStdout for events, something like onEvent.

Looking forward to see where this is going with my feature request :)

mxschmitt commented 4 months ago

Where do you currently emit the events from inside your worker?

kbrueckner commented 4 months ago

My library internally works with events. Currently whenever an event is caught, it turns it into a message to stdout. This all happens within the worker itself.

Instead of writing to std I would like to pass the message on to the IPC to main so that I can then catch it there.

If it helps we could have a quick chat on the playwright discord server. I had opened a help request there initially. (https://discord.com/channels/807756831384403968/1257555666005131328)

kbrueckner commented 4 months ago

Maybe you could offer something like sendCustomEvent (IPC) on the worker and in the reporter onCustomEvent?

kbrueckner commented 2 weeks ago

Is there anything more you would need? This ticket is in collecting feedback state for quite a while now.

microsoft / playwright