Cause jasmine framework to broadcast suite and test start and end events to the runner process

[object88]

I am working on a new reporter, which will output messages on a per-suite basis. This will reduce confusion between messages coming from different runners, which are typically and frequently intertwined.

One of the key features is the ability to emit a message from a suite (from a logging class, for example), and have it reported inline with the test or setup that it gets invoked from. For this to work, however, the logger needs to know what the current suite and test is. The existing wdio-jasmine-framework shares this information, but only with reporters, which are running on the originating process.

This PR will broadcast [suite|test]:[start|end] to the currently running process as well.

[christian-bromann]

This PR will broadcast [suite|test]:[start|end] to the currently running process as well.

How does this help you if the reporter you are writing also lives in the originating process?

[christian-bromann]

How will your reporter be different from the spec reporter?

[object88]

How will your reporter be different from the spec reporter?

Great question. When a single runner is executing, there will effectively be no difference from the spec reporter, because the spec reporter instantly console.logs the suite and tests's success and error messages, and those specs and suites are run synchronously (if I understand / remember its behavior correctly). However, when you have many runners executing at the same time, the output messages can become jumbled, and it's not at all apparent what messages are connected to which test or suite. This is the problem my new reporter aims to address. This is especially useful when running on a CI process.

How does this help you if the reporter you are writing also lives in the originating process?

I likely am not doing a good job of describing my work; my apologies. The reporter is running in the originating process, and it will receive coherent:message events (just as it receives suite:start, test:end, etc). The coherent:message events include a reference to the current suite and test, so that the reporter knows which spec and test to output the messages with, once the suite is completed. However, for this to happen, the logging class -- running as a child process -- needs to know what the current suite and test is. At the moment, that information is private to wdio-jasmine-framework. Broadcasting that information locally to the runner will allow a logger to know that information, so that it can be included on a coherent:message event.

[christian-bromann]

the output messages can become jumbled

Not really. The spec reporter outputs the suite result at once when the suite finishes.

The reporter is running in the originating process

Does that mean you initiate it in your test file?

[object88]

The spec reporter outputs the suite result at once when the suite finishes.

You are correct -- I was thinking about the cucumber reporter. However, the spec reporter does not work for WDIO v4, if I am reading the issues correctly.

Does that mean you initiate [the reporter] in your test file?

No, the reporter is initialized the same as ever, i.e., it's declared in the wdio.conf.js file, and is instantiated as usual.

[christian-bromann]

if I am reading the issues correctly.

Yes, but I started working on it and will release it soon when I have a first working version

No, the reporter is initialized the same as ever

Well in this case the reporter will life in the originating process and doesn't need that change

[object88]

I am glad to hear that the spec reporter work is started; I finished mine before there was any announcement of this effort. However, the spec reporter still does not log messages which aren't test and suite progress messages. My reporter does this -- if the wdio-jasmine-framework would emit enough information for a logging object (running in the test itself) to know which test and suite is executing. This functionality could be included in the spec reporter, if you like, but it would still necessitate the proposed change to wdio-jasmine-framework.

[christian-bromann]

still does not log messages outside of the test and suite progress

What kind of messages do you mean?

[object88]

Debugging messages, for example. Anything that a test writer may want to put into a [after|before][All|Each] block, or it block to help decipher a problem with a test.

[object88]

(Also, I realize now that the title of this PR is terribly misleading -- will correct that.)

[christian-bromann]

Anything that a test writer may want to put into a before[All|Each] block, or it block to help decipher a problem with a test.

Can you give an example?

[object88]

An excessively trivial example is in the README for the reporter project.

A more interesting would report timing issues; my team has had trouble with wdio & selenium, writing helper methods which would wait for DOM elements, etc. While in the middle of the learning process, we would make mistakes by incorrectly using async/await, or returning from a browser request prematurely. By adding logging messages, we were able to determine problems that led to early afterEach invocations, etc.

[christian-bromann]

@object88 why do you need the reporter for that if you just can access the logging API like browser.logger.info("some message")?

[object88]

I have tried calling browser.logger.info (which I understand is this), and this does not solve my problem. It also writes directly to console.log (after decorating the message), and so, messages cannot be correlated to tests when you have multiple runners and/or are delaying the suite completion output. You will find the same issue when you complete your re-write of the spec reporter, as you are looking to delay the test reports until after a .spec.js file is complete.

Let me give you a more concrete example, with idealized output from a working spec reporter. Let's say that you have three runners, one for firefox, one for safari, and the other for chrome. You have a test spec that looks something like this:

describe('my magic computer', () => {
  beforeEach(() => {
    if (...) {
      browser.logger.info(`browser.getElement('foo') is ${x}; browser.getElement('bar') is ${y}`);
    }
  });
  afterEach(() => {
    if (...) {
      browser.logger.info(`z is ${z}`);
    }
  });
  it('tests adding', () => {
    browser.waitUntilVisible(baz);
    if (...) {
      browser.logger.info(`Failed to find element`);
    }
  });

})

If this is run with a spec-like reporter, and two browsers are failing, this is the kind of output you might see:

Failed to find element
browser.getElement('foo') is {a:1,b:2}; browser.getElement('bar') is {c:15}
Failed to find element

Suite "my magic computer"
running on chrome
X test adding
✓ test subtracting

Suite "my magic computer"
running on firefox
X test adding
✓ test subtracting

Suite "my magic computer"
running on safari
✓ test adding
✓ test subtracting

How do you know what messages came from what runner instances?

[christian-bromann]

See wdio spec reporter output:

It prepends the capability and the id of the runner to its output that should help you to identify the browser

[object88]

That is a very simple test case, with one runner, and no logging messages -- those messages all come from runner status messages which inherently are aware of the browser (via the runner capabilities). The class you mentioned (browser.logger) is unaware of the current runner, and so cannot produce the same output. The current code for the spec reporter (which I understand that you're working on updating) does not have any capability to inject debugging messages that are not part of the test/suite/runner start/end variety.

There is value in being able to thread logging messages with the suite outcome -- this gives context for why tests are failing. Currently, logged messages typically don't know what test or suite they are running in, and certainly do not know which runner or browser. Dealing with issues of flakiness and variations between browsers, Selenium drivers, and SauceLab configurations makes debugging very difficult, and this can help alleviate that pain.

[christian-bromann]

Let's close this. I think this should be solved differently that works across all frameworks and reporters.